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INTERACTIVE DATA VIEW AND COMMAND SYSTEM 
Field of the Invention 

The present invention relates to an interactive data view and command 
system including a wearable signal input unit providing high level of wearer 
comfort, perfonnance and flexibility. 

Background of the Invention 

Recording, processing and reproduction of images grows with an 
increasing demand for infonnation and their clear graphic visualization. 
Electronic image processing also involves processing of images taken by 
cameras, scanning systems and sensors in the visible light spectrum as well as 
in other sections of the electromagnetic spectrum such as the infrared, the radio 
frequency and the X-ray frequency range. After electronic processing, many 
images are reproduced either as individual images or as moving Images on an 
image reproduction screen such as a display for presenting the information to the 
eye. 

There are special image contents easier recognizable by electronic image 
processing using, for example, local frequency filtering, margin sharpness 
increasing. Image data compression, image correlation, dynamic reductions and 
false color-coding. On the other hand, other techniques are concerned with the 
superposition or subtraction of auxiliary images taken from different spectral 
ranges or with the superimposing of stored plans, maps, and drawings onto the 
original image. 

By applying image processing, the infonnation content of the actual, direct 
image can be intentionally increased or reduced. Image processing is used in a 
wide range from increasing the image contrast to blend ing-in of additional 
information, marking of details, and highlighting dangers. In many of these 
applications, it is disadvantageous that the electronic camera is a "second eye 
system" separate from the human eye. This disadvantage is due to the fact that 
the images are seen from another recording location and that additionally, the 



pictures on the image screen are presented at another observation location than 
the eye. Thus, the human eye must constantly change between the direct 
observation and the indirect observation while taking Into account different 
observation angles, different image details, and different size ratios which leads 
5 to physical impainnents and delays when decisions must be made. 

These problems may be solved by the "head-up-display" (HUD) technique 
used in the piloting of combat aircraft, in that Important information such as 
instrument displays and target data are inserted or fade-In Into the open 
spectacles of the pilot's helmet and thus into the visual field of the pilot. This 
10 technique is also used experimentally in the automobile industry for displaying of 
instrument readings on the windshield so that the driver is not distracted from 
viewing the road by viewing the instrument panel. 

The "virtual reality" or "cyber space" techniques use closed spectacles (i.e. 
glasses) where the outside view is blocked and three-dimensional full Images are 
15 projected into the eye with virtual reality. These virtual reality images are then 
modified in response to body motions such as locomotion, movement of an arm, 
a finger, or head and eye movements. 

There are other HUD techniques and image detection and projection 
techniques described in the PCT Application PCT/EP97/04188 (published as 
20 WO98/05992) and U.S. Patent 6,227,667. The apparatus described in these 
documents capture (or detect) the retinal reflex Image and also enable a 
superimposltion of supplementary images in the eye. 

There is still a need for an information system capable of providing 
interactive data view and command applications for use in various environments. 



25 



Summary of the invention 

The present invention is a method and infonnation system for capturing 
signals, for processing signals, and for providing signals at least partially based 
on, or bearing correlation to, the captured signals. The information system 
30 includes a signal input unit, a wireless communication unit, and an output unit. 
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The signal input unit (preferably an optical signal unit) is constructed and is 
positionable to capture signals associated with an eye. The output unit is 
constructed to provide information based on the captured signals or to provide 
information as a function of the captured signals or in correlation with the 
5 captured signals. 

The signal input unit may be a non-optical signal unit for capturing non- 
optical signals (e.g., acoustic data, electric data, magnetic data), or an optical 
signal unit for capturing optical data (e.g., ultraviolet, visible or infra-red). The 
optical signal unit can capture data by scanning or large image detection using 

10 light sensitive arrays WKe CCDs. In some embodiments, the signal input unit may 
be integrated with the output unit, and may also include an information unit. 

Preferably, the optical signal unit and possibly other units are constructed 
in the form on spectacles or goggles worn by a user, as described in the PCT 
Application PCT/EP97/04188 (published as WO98/05992) and U.S. Patent 

15 6,227,667; U.S. Application 09/462,440; or PCT Applications PCT/EPOO/09840, 
PCT/EPOO/09841 and PCT/EPOO/09843, filed on November 6, 2000; the content 
of these applications is explicitly incorporated by reference into this application as 
if fully provided herein. The various embodiments and units described in the 
above patent documents have been further improved to create economical, user- 

20 friendly, comfortable, high performance devices. In particular, adjustments and 
improvements of the aforementioned optical units provide a novel manner of 
projecting images onto or in front of the user's eye, including the retina and other 
surfaces. The present optical signal unit and output unit are not limited to scan- 
like capture of the retinal reflex image or projection on the retina, but utilize other 

25 properties and function of the eye and human perception. 

Since the apparatuses and systems described in the aforementioned 
applications are preferably embodied in the shape of spectacles, they will also be 
designated hereinafter, for the sake of simplicity, as a spectacle system, but this 
designation does not and should not imply any limitation. The optical apparatus 
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that scans the retina and displays corresponding image modifying or image 
improving optical information onto the retina are optimally used. 

The signal input unit may be an optical signal unit is constructed to capture 
visible signals, infrared signals or ultraviolet signals. The output unit may be 
5 constructed to provide the information on a display by projection or using a 
graphical user interface. The output unit may be constructed to project the 
information. For example, the output unit is constructed to project at least part of 
the information onto the retina of the eye, or the output unit is constructed to 
project at least part of the information onto a two or three dimensional surface 
10 remotely located from the eye. The output unit may be constructed to provide 
the infonnation in an audible format. The output unit is constructed to provide the 
if infomnation in a format causing sensory skin stimulation, stimulating tactile 

m organs, or visual, audible, smellable, or tastable senses. 

^1 According to another aspect, an information system includes an optical 

|j 15 signal unit, a wireless communication unit, and an output unit. The optical signal 
= unit is constructed and positioned to capture signals associated with an eye. The 

output unit is interfaced with the wireless communication unit and is constructed 
m and an-anged to provide information using a con-elation unit. The correlation unit 

p is constructed to find suitable relationship between the captured signals and 

20 additional data. 

Preferred embodiments of this aspect include one or more of the following 
features: 

The information system may also include an information unit. The 
information unit may include various databases, processors, sensors, an 
25 information network connection, an evaluation unit, and other devices providing 
additional data to the information system. 

The correlation unit is constructed to determine a presentation relationship 
of the captured signals and the additional data. The correlation unit is 
constructed to determine the presentation relationship between the captured 
30 signals and the additional data in terms of location of the additional data with 



- 4 - 



respect to the captured data. Alternatively, the correlation unit is constructed to 
determine the presentation relationship between the captured signals and the 
additional data in terms of presentation timing of the additional data with respect 
to the captured data. 

The correlation unit may be constructed to determine the presentation 
relationship between the captured signals and the additional data in terms of 
relative color display of the additional data with respect to the captured data. 
Alternatively, the correlation unit is constructed to determine the presentation 
relationship between the captured signals and the additional data in terms of 
relative size of the additional data with respect to the captured data. 

In short, the con-elation unit perfomis the correlation funtion that may be 
"one-to-one" overlap of a scanned and a projected Image, or may completely 
differ from the "one-to-one" overlap, or may include different type of data. For 
example, the input signal unit may include a microphone to the external system, 
in particular a directional microphone, which is directed as a function of the head 
position or the gaze direction to allows a further sensory dimension to be 
realized. 

The correlation unit may direct projection of a correlated field-of-view 
image of an individual (e.g., a fireman) or a mosaic of several Images Into the 
eye or on a projection surface of another individual (e.g., a fire chief overseeing 
the operation). The correlation unit enables, for example, the projected image of 
the entire scene (field-of-vision) to be partially overlapped by data symbolizing 
the position and severity of the sighted or otherwise captured (or detected) event 
(e.g., fire location and temperature colored in accordance with the strength of the 
fire, which could be blended Into each fireman's view). Alternatively, a disruptive 
effect of the natural field of view not necessarily covered up by the projection of 
information can be reduced via a complementary-color "wiping out" in which 
complementary-colored pixels are determined on the basis of the light captured 
from the field of view whose correlated projection onto the respectively 
associated regions of the retina make the natural background appear white on 



account of the addition of colors. The virtual position of the fixation fixed, e.g., 
via a fixation with the eye in conjunction with a blinking of the eyes (or a pressing 
of a button or even automatically), for example, by an image-processing 
evaluation of the field of vision that determines an area of the field of vision 
having as little content as possible. 

According to yet another aspect, an information system includes an optical 
signal unit, an infonnation unit, a wireless communication unit, and an output 
unit. The optical signal unit is constructed and positioned to capture signals 
reflected back from at least one eye comprising the retina. The field-of-view 
capturing unit Is constmcted and arranged to capture light from a field of view 
associated with the retina without capturing a retinal reflex image. The output 
unit Is constructed to provide Infomnation, at least partially obtained via the 
communication unit, in cooperation with the information unit as a function of the 
captured light and in correlation with the captured signals. 

According to another aspect, an infonnation system includes an optical 
signal unit, an information unit, a wireless communication unit, and an output 
unit. The optical signal unit is constructed and positioned to capture signals 
reflected back from at least one eye comprising the retina. The optical signal unit 
comprises a scanning detection unit constructed to at least partially capture a 
retinal reflex image of the retina. The output unit is constructed and an-anged to 
provide infomnation, at least partially obtained via the communication unit, in 
cooperation with the information unit as a function of the captured signals. 

The output unit may be cooperatively constructed and arranged with the 
optical signal unit to enable projection of information onto the retina. 
Alternatively, the output unit is not constructed to project infonnation onto the 
retina. 

According to yet another aspect, an infonnation system includes an optical 
signal unit, an information unit, a wireless communication unit, and an output 
unit. The optical signal unit is constructed and positioned to capture signals 
reflected back from at least one eye comprising the retina. The optical signal unit 



comprises a scanning detection unit constructed to at least partially capture a 
retinal reflex image of the retina during a first scanning operation and carrying out 
a less comprehensive capture of the retinal reflex image during a later scanning 
operation. The output unit is constructed and an-anged to provide infomiation, at 
5 least partially obtained via the communication unit, in cooperation with the 

information unit as a function of the captured signals. The output unit comprises 
a scanning projection device constructed to project at least part of the information 
onto the retina. 

According to yet another aspect, an information system includes a signal 

10 input unit, an infomiation unit, a wireless communication unit, and an output unit. 
The signal input unit is constructed not to capture any signals reflected back from 
the retina. The output unit Is constructed and arranged to provide information, at 
least partially obtained via the communication unit In cooperation with the 
information unit as a function of the captured signals. The output unit comprises 

15 a scanning projection device constructed and arranged to project at least part of 
the information onto the retina. 

According to this aspect, the signal Input unit may be an optical signal unit 
constructed and positioned to capture signals reflected back from at least one 
eye but not from the retina. 

20 According to yet another aspect, an information system includes a signal 
input unit, an information unit, a wireless communication unit, and an output unit. 
The signal input unit is constructed and positioned to capture at least two types 
of signals wherein one or both signals may be reflected back from the eye. The 
output unit constructed and arranged to provide Information, at least partially 

25 obtained and/or provided via the communication unit. In cooperation with the 
information apparatus as a function of the captured signals. The output unit 
comprises a scanning projection device constructed to project at least part of the 
information onto the retina of the eye. 

Preferred embodiments of the above aspects Include one or more of the 

30 following features: 
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The Information system includes a spherical or spherical-acting reflection 
layer operably positionable at a location immediately anterior and substantially 
oonfocal to the eye, and the optical signal unit is constructed to capture optical 
fleld-of-vision signals reflected off the spherical or spherical-acting reflection 
layer. 

The field-of-view capturing unit is constructed to capture visible light from a 
field of view associated with the retina without capturing a retinal reflex image 
thereof. The output unit is suitable for providing the information in correlation 
with the captured visible light. 

The information unit comprises an evaluation module constructed to obtain 
Image information with regard to the field of view from the captured visible light. 

The output unit comprises a projection device constructed to project the 
image information onto the retina in conrelation with the captured signals such 
that a naturally perceived field of view and projected image infomiation are 
perceived as a unitary image by the retina. The scanning and projection may 
utilize separate beam paths. 

The information system provides a function that encompasses a temporal or 
spatial correlation between the provision of information and the captured visible 
light. This function may also encompasses pattern recognition that yields at least 
one infomiation key, and the information keys serve for an information query 
based on the information apparatus. 

The optical signal unit includes a scanning device that records an at least 
partial capture of a retinal reflex image of the retina in a first scan operation and 
candies out a less comprehensive capture of the retinal reflex image in a later 
scan operation. The optical signal unit may capture a retinal reflex image of the 
retina only partially or not at all. 

The field-of-view capturing apparatus includes a spherical or spherical-acting 
reflection layer suitable for deflecting a portion of a light directed at the eye into a 
sensor apparatus for capture. The field-of-view capturing unit or the optical 
signal unit may be suitable for at least partially capturing a corneal reflex image 



of the eye. The optical signal unit or the field-of-view capturing unit may be 
fabricated as portable units. The information unit may include a databank 
(including various storage devices), a sensor system, an information network 
connection and/or an evaluation unit. The infomnatlon unit may be fabricated as 
5 a portable unit. 

In accordance with one aspect of the optical input unit, the optical 
apparatus is wearable by an operator and capable of scanning the retina up to 
frequencies of 100 Hz and an image falling onto the eye, in particular a retinal 
reflex image. The optical input unit is a part of an interactive data view and 

10 command system, with the particular advantage that, for operating the system, a 
least possible effort on the part of the operator is necessary. The information to 
be queried can be transmitted to the operator with shortest possible delay either 
in the forni of Image signals and/or in the form of signals for controlling further 
information reproduction devices that can be operated e.g. on an acoustic or 

15 other sensory basis. A system thus results that is distinguished by a highest 
measure of directness of the interactive information exchange. 

The data transmission unit may include a mobile communication system is 
switched between the data view and command system and an external 
Information source. In this manner, the field of application of the entire 

20 information system is additionally extended. The data transmission unit can 
include a mobile telephone or a computer, e.g., a laptop or palmtop having a 
suitable data remote transmission interface. 

The system can execute additional control commands that control the flow 
of information. These control commands can be based on optical information 

25 signals that are output by the optical apparatus. For example, a menu bar is 
blended in to the operator into the field of view by outputting image signals onto 
the retina via the optical apparatus, again with an image frequency of roughly up 
to 100 Hz. The selection of a menu item is carried out either via a control mouse, 
or instead solely via the focusing of this selected menu item and with the aid of a 

30 selection signal. The selection signal can be a button, voice command or an eye 
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blink signal. The focusing can be determined optically since the selected menu 
item is situated in the center of the fovea centralis, which can be determined, in 
parallel, while the scan of the retina / of the image falling onto the eye is mnning. 
The signal input unit may include a microphone and a speech analysis 

5 unit. 

The operator's additional control commands can be based on acoustic 
information signals that are output from a speech input unit. Since, the signal 
input unit may include, an optical apparatus is used that cyclically scans the 
operator's retina, and the system can be employed to carry out an active 

10 operator recognition in a highly economical fashion. It is thus possible, on the 
one hand, to automatically protect the system / system operations from 
unauthorized use and, on the other hand, to carry out an automatic personalized 
set-up of the system to the operator wearing the system. For this purpose, it is 
solely necessary to temporarily store the retina structure captured by the optical 

15 apparatus in the form of a data record in the signal processing apparatus and to 
compare it to a person-specific data record already stored. 

Due to the compactness and the short signal paths from the optical 
apparatus to the eye, on the one hand, and to the communication apparatus on 
the other hand, application of the system is quite versatile including the use in the 

20 realm of medicine, in particular In the field of ophthalmology, as both a 
therapeutic and analytic device. 

Additional applications are described in the PCT Applications PCT/EPOO/09840, 
PCT/EPOO/09841 and PCT/EPOO/09843, which are incorporated by reference. 
The present information system includes numerous embodiments that provide 
25 one or more of the following features: 

- at least partially captures a cornea reflex image of the eye; 

- directs a part of the light falling onto the eye into a sensor apparatus by using 
the spherical or spherical-acting reflection layer; 

- determines the retinal image via the degree of oxidation of the retinal cones 
30 and/or the retinal rods; 
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- carries out solely a partial capture of a retinal reflex image; and/or 

- comprises a field-of-view capturing apparatus that captures visible light from 
the naturally perceived field of view without capturing a retinal reflex Image. 



5 At the same time, the present system in accordance with the described 

embodiments can provide: 

(a) information as a function of a naturally perceived field of view of a person; 

(b) information as a function of signals captured from an eye, yet does not 
project these into the eye from which the signals were captured; or 

10 (c) information as a function of signals captured from an eye, wherein the 

infonnation is at least partially projected into the eye, the signals, however, are 
not captured in the manner known from the PCT Application PCT/EP97/04188 
(published as WO98/05992) or U.S. Patent 6,227,667. 

The signal input and output units, including the described spectacle 
15 systems, are based on a combination of several of the aforementioned 

fundamental concepts, the result of which is a natural interrelationship of the 
associated three applications. The present information systems allow additional 
information to be provided to the operator that goes beyond our personal 
knowledge and sensory impressions. Examples of this include searching for an 
20 electrical cable under the plaster in a wall, navigating in an unfamiliar city, 

collecting wild mushrooms, and inspecting a possibly dangerous object using a 
remote-controlled robot. The present information system can provide information 
to a person or another system. 

The strong dependency of a person having vision on their visual senses 
25 clearly contributes to the difficulty of providing additional information. Indeed, the 
fact that people having vision primarily use their eyes makes it necessary, in 
many cases, to either input the supplementary information via the eyes or to 
determine the supplementary information based on the information seen. In the 
case of input via the eyes, however, the orientation of the eyes must be taken 
30 into exact consideration for correct "placement" of the input information and to 
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avoid a "jittering" or "blurring" thereof. In addition, in many cases the information 
should be made available without a controlled movement of the eyeballs; a car 
driver may have a map on his lap but would prefer not to have to look away from 
the street. 

Due to their dependency on solid media, e.g. paper, CRT and LCD 
screens, etc., prior visual information systems have not been in a position to 
sufficiently fulfill the comfort needs of a person having vision. Non-visual 
information systems previously lacked the correlation to that which is seen that is 
natural for people having vision. 

Through the incorporation of the interactive data view and command 
system into an information system in accordance with the dependent claims, a 
system is provided whose presentation of information fulfills the natural needs of 
person having vision in a previously unachieved manner. At the same time, the 
information system is improved over the prior art with regard to its 
implementability and economy. 

In its most general form, the information system in accordance with the 
invention comprises a signal capturing apparatus that captures signals reflected 
back from an eye comprising a retina, an information apparatus and an output 
apparatus that provides, in cooperation with the information apparatus, 
information in correlation with the captured signals. Preferably, the information is 
provided as a function of the captured signals and/or as a function of visible light 
captured from the naturally perceived field of view. 

Preferably, one of the aforementioned spectacles in which a scanning 
detection apparatus at least partially captures a retinal reflex image of the retina 
serves as a signal capturing apparatus. A modification of this detection 
apparatus that captures light reflected on a cornea of the eye in lieu of the retinal 
reflex image is particularly advantageous for infrared applications since the 
cornea strongly reflects light having a wavelength of roughly 1.1 |jm. It is also 
fundamentally possible to make correspondingly valuable statements about the 
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image falling onto the retina by capturing the chemical change of the rods and/or 
cones. 

The present system uses the advantages of capturing the field of view 
complementary to the capturing of signals reflected back from the eye. For the 

5 purpose of such a complementary capturing, the field-of-view capturing 
apparatus and/or the information apparatus of the information system in 
accordance with the invention preferably comprises a spherical or spherical- 
acting reflection layer that is positioned essentially confocal to the eye that 
deflects a part of the light directed onto the eye into a sensor apparatus for 

10 capture. Due to the fact that the reflectivity of the reflective layer is several times 
higher than that of the retinal or corneal reflex, considerably more light can be 
captured using equally sensitive photo-sensors. Moreover, correspondingly 
cheap photo-sensors could be used in the sensor apparatus. It can also be 
advantageous if the light falling onto the eye is not solely, only partially or not at 

15 all captured via the retinal reflex. 

Depending on the intended application, not all spatial regions of the field of 
view must be captured. In an application, for example, in which supplementary 
information with regard to an object upon which the eye has fixed its gaze is 
provided by an information system in accordance with the invention, it could be 

20 sufficient to capture the light falling onto the fovea and to subject it to a pattern 
detection or other type of analysis since an object upon which the eye has fixed 
its gaze is typically imaged on the fovea which represents the area of keenest 
sight. Accordingly, capturing the light that falls onto this part of the retina would 
possibly be sufficient to determine a sufficient number of characterizing object 

25 features. 

It is also sensible if only a limited spectral range of the light falling onto the 
eye is detected. If, for example, the infrared light falling onto an eye is detected, 
the orientation of the eye and/or valuable information from the field of view can 
be determined, even at night. 
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Accordingly, any limitations with regard to the capturing of the light falling 
onto an eye can be meaningful. In particular, limitations of the captured spectral 
range, the captured regions of the field of view and/or the captured time spans of 
vision are applied as necessary. 
5 For the purpose of redundant or stereoscopic image capture, the 

correspond apparatus of the information system in accordance with the invention 
can be designed so as to capture the light falling onto several eyes. Depending 
on the field of application, the eyes must not necessarily belong to a single 
person. For example, it would be possible to display the images perceived by 

10 the eyes of several firemen onto monitors in a command center in addition to 
position and fire strength information determined from an infrared spectral 
analysis of the images. 

In the field of ophthalmology, a distinction is made between the terms 
"field of view" and "field of vision." A field of view is the part of a space that can 

15 be seen with a stationary eye. A field of vision is the region that can be seen with 
the eyes. Consequently, here, as elsewhere, the field of view is to be understood 
as the cause of the light that naturally falls onto an eye. 

The present system may use a field-of-view capturing unit (i.e., field-of- 
view capturing apparatus) for capturing light from the field of view associated with 

20 the eye of a particular quality, /.e. with a sensitivity, a resolution, a sharpness, 
etc. This field-of-view capturing of visible, infrared or ultraviolet light far exceeds 
the natural visual acuity of the eye. The field-of-view capturing unit is not limited 
to capturing of the field of view, but may include partial or complete capturing of 
the field of vision that encompasses an at least partial capturing of the field of 

25 view. 

The field-of-view or field-of-vision capturing units capture high quality 
images that can also serve as a basis for an extrasensory presentation of 
information. For example, field-of-view information can be obtained from the 
captured light of the field of view and projected onto the retina such that the 
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image seen by the eye seems at least partially sharper, closer, wider angled or in 
some other manner extrasensory. 

The present system enables novel ways of correlating and presenting data 
acquired, computed (manipulated), received via a wireless communication unit, 

5 or from a storage device. Importantly, the provision of information with the 

signals captured by the signal capturing unit, the system treats the corresponding 
parts of the captured light during a processing that occurs in the course of the 
provision of information as if they were reflex images captured from the eye, i.e., 
as if they were that which is truly seen. The information system can also 

10 combine high quality field-of-view information directly from the truly seen field-of- 
view infomnation from the eye. 

The correlation of the presentation of information with the captured signals 
reflected back from the eye can be carried out, for example, by capturing several 
pixels of an ocular reflex image, e.g. a cornea or retinal reflex image, that are 

15 brought into connection with corresponding pixels from the captured field of view 
via an evaluation apparatus. A gaze direction of the eye determined via the 
captured signals can also serve to establish a correlation between the field of 
view information obtained from the captured field of view and which is truly seen. 
As will be described herein below, the correlation can, however, also comprise 

20 projecting obtained field of view information onto the retina in a correlated 
manner to that which is seen. 

The information unit may include a data bank, a sensory unit, an 
information network connection, one or several processors, and/or an evaluation 
unit. The term "evaluation unit or apparatus" means any type of evaluation 

25 apparatus and in particular image processing devices. The information unit can 
include one or several sensory units collecting data tactually, visually, audibly, 
smellably and/or tastably. 

The sensory unit provides an extrasensory perception in connection with 
the visible data. For example, when searching for an electric cable, the sensory 

30 unit can include one or several magnetic field sensors capable of localizing 
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metallic cables with regard to a known coordinate system, e.g., the captured 
field of view. Image processing software superimposes sensor signals from 
existing electrical cables and provides a supplementary image, as described in 
aforementioned patent applications, projected onto the image seen by the eye. 
5 The information unit may also include other types of sensors used as an 

information source, in particular when the sensor is activated and/or queried on 
the basis of the captured light image. For example, during inspection of an 
integrated electronic circuit the position of a conductor on a manufactured chip 
could be computed after a directed gaze at that conductor on a circuit plan of the 
10 circuit and the pressing of a button so that the current and voltage values of the 
conductor are determined using the non-contacting measuring device and 
y presented to the use via the spectacle unit. 

m The information unit may also include a data bank and an information 

25 network connection. For example, is an intra-company mail distribution system 

O 15 files include bar code stickers that uniquely identify the respective file. If a file is 
V" to be sent within the company, the sender enters e.g. the receiver's extension 

2 and a code designating the file using software that correspondingly stores these 

U data in a data bank in one of the many known ways. When the file is later sorted, 

n the identifying bar code is captured via the spectacle unit worn by a mail 

20 distribution employee, e.g., via a directed gaze and a push of a button, and 
recognized via a recognition apparatus or recognition software. The data 
associated with the file relevant to mail distribution are retrieved from the data 
bank via a radio connection to an intra-company data network and these data are 
presented to the mail distribution employee, after pre-processing, if necessary, 
25 via a suitable output apparatus, e.g. as an announcement via headphones "Mr. 
Schmidt, finance department, building G, second floor, room 310." 

The presentation of information that complies with the needs of a person 
having vision in a manner not previously achieved. This can comprise providing 
the information to the person in a suitable manner, i.e. using one or more of the 
30 five senses. The information can, however, be presented in an arbitrary manner 
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and does not require a particular addressee. For example, the infornnation can 
be provided to a further system or radiated into the environment via an optical or 
acoustic output apparatus. The claimed dependency between the provision of 
information and the light image falling onto the eye guarantees that the 
5 correlation that a person having vision expects exists between the infornnation 
provided and that which is seen. 

This dependency is taken into consideration during the determination of 
the information, during the provision of the information or during both of these 
inherent processes. Examples for establishing this dependency during the 

10 determination of the information are given above. During the provision of 

information, this dependency can be established, for example, by blending the 
infonnation into the image seen by projecting back into the eye in such a way 
that a temporal, spectral, spatial, contractual or other sensible correlation is 
established between the information and the image seen. In particular, the 

15 dependency can consist of the captured light image being used to determine the 
position and orientation of the eyeball so that an image projected onto the eye for 
the sake of providing the information appears to stand still during a motion of the 
eye, appears to move with the eye during a motion of the eye or appears to move 
in accordance with a predetermined course during a motion of the eye. In 

20 particular, the effect of the saccadic motion of the eye on these processes can be 
taken into consideration and/or compensated. 

The optical signal unit or the output unit (or the spectacle apparatus) can 
determine the position and orientation of at least one eye quickly, accurately and 
at little expense, e.g. at a detennining rate of 100 Hz, a positional accuracy of 

25 several micrometers and using a portably constructed apparatus. By using the 
information during the dynamic evaluation of the orientation of the eye, the 
processing can be carried out so quickly that the accuracy is not impaired by the 
saccadic motion of the eye. This is achieved by a signal-capturing unit, which 
does not contact the eye and captures signals reflected back from the eye. 

30 Reflectable signals, e.g., sound or electromagnetic signals, allow a high 

- 17 - 



frequency capturing such that the processing speed is primarily determined by an 
evaluation unit. The evaluation unit may have, depending on the application, 
very high processing speed, low power consumption and small system size. 

The information system may itself serve as a reference coordinate system. 
5 However, the information system may solely represent an intermediate reference 
coordinate system in another reference coordinate system and that the 
relationship between the intermediate reference coordinate system and the 
reference coordinate system is determined e.g. via the evaluation apparatus or 
another mechanism. 

10 The signal input unit may include an optical signal unit that captures light 

reflected back from the eye. Light is an excellent medium for transmitting the 
signals reflected back from the eye since the presence of light is a prerequisite 
for the ability to use the eye. However, the ocular reflex signal information that 
results through the reflection on the eye is superimposed with field-of-view signal 

15 information transmitted by the light from the field of view. These differing pieces 
of information, however, can be distinguished through use of known signal 
processing methods and can be sensibly used for determining the orientation of 
the eye. This is particularly true when the signal transmission medium is from a 
signal source belonging to the information system that applies a predetermined 

20 signal to the medium prior to its reflection on the eye. 

The signal input unit may also capture signals from other signal 
transmission media. Components for generating and capturing sound waves, for 
example, are commercially available in various cost-efficient and compact forms. 
Such components can also be implemented as integrated elements of an 

25 integrated circuit. Similar considerations apply to the non-visible frequency range 
of electromagnetic waves. 

The signal input unit may also capture signals from different media or 
spectral ranges thus providing improved system characteristics. This is based on 
considerations that the evaluation unit takes over other system tasks in the case 

30 of underload and that the signal processing carried out by the evaluation unit 
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depends strongly on the information content of the signal to be processed. The 
information system can use signal capturing that only demands a little 
performance from the evaluation unit, but itself might not supply the basis for 
sufficient accuracy and to complement and/or calibrate this low-processing signal 
5 capturing via the results of an accurate and processing-intensive, yet only 
intermittently executed signal capturing such that the necessary accuracy is 
achieved at any time. 

The capturing of the retinal reflex in which the retinal reflex of natural or 
artificial light is intermittently or partial captured as the signal reflected back from 

10 the eye has turned out to be useful. A full capturing of the retinal reflex is both 
time consuming and demanding on performance. On the other hand, a capturing 
of the retinal reflex is useful inasmuch as it allows the relationship of the 
perceived field of view to the retina to be directly determined. Indeed, as noted 
above, a processing of the captured retinal reflex allows both retinal features 

15 such as e.g. the fovea centralis or the blind spot as well as the reflex image of 
the light falling onto the eye to be determined. The network of blood vessels 
present in the choroid coat also becomes visible through appropriate processing 
of the retinal reflex image, which yields a very good basis for detemiining the 
orientation of the eyeball. If the retinal reflex is thus captured intermittently or 

20 partially, the processing complexity can be reduced without sacrificing an exact 
determination of the relationship of the perceived field of view to the retina. 
Naturally, the retinal features can be followed without capturing the retinal reflex. 
For example, the blood vessels of the choroid coat can be recognized via their 
radiation of heat that is visible in the infrared range. 

25 The above system may be used for the analysis of a patient's sight, 

wherein a predetermined pattern or a predetermined distribution of patterns is 
generated on the retina or on selected regions of the retina using the projection 
unit. The system may use movement patterns and/or the noise fields and/or the 
spatial vision of a patient's eye, wherein random dot patterns are generated on 

30 the retina using the projection unit for test purposes. The system may be used 
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for determining anomalies in the motor response of tlie eyeball, wherein a unit for 
determining and monitoring the position and/or orientation of the eyeball is 
integrated into the system. The system may be used for detecting 
parasympathetic/ sympathetic efferences, wherein the motor response of the 
pupil is monitored and evaluated by means of a detector device. The system 
may also be used as one or more of the following: a synoptophor or 
synoptometer with no device convergence, a device for determining 
cyciodeviation, a phase difference haploscope, a device for detecting phoria 
identical to the visual axis with different lines of sight, for checking the function of 
the retina, making use of a sample electro-retinogram (ERG) and a correlation 
device, with which an image played onto the retina can be brought into 
correlation with the ERG actually determined, for measuring the contrast 
sensitivity of a patient's sight, preferably as a function of the spatial frequency, for 
white-noise-field campimetry, for determining the extent and the position of 
central field of vision defects (scotomae), as a VEP (visual enabling for precision 
surgery) device, or as an SLO (scanning laser ophthalmoscope) device. 

Brief Description of the Drawings 

Figure 1 shows schematically an Interactive data view and command 
system in accordance with a first embodiment of the invention. 

Figure 2 is a detailed view of an eye in cross-section. 

Figure 3 shows schematically an embodiment of interactive spectacles 
employed in the invention in which an optical signal unit is provided in the form of 
a scanning eye detection apparatus. 

Figure 4 shows schematically is an embodiment of interactive spectacles 
employed in the invention in which an output apparatus in the form of a scanning 
projection apparatus is provided. 

Figure 5A shows schematically interactive spectacles in accordance with a 
fourth embodiment. 
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Figure 5B is a detailed drawing of a combined signal capturing and 
projection unit illustrated in Figure 5. 

Figure 6A shows schematically interactive spectacles in accordance with a 
fifth embodiment. 

5 Figure 6B is a detailed drawing of a combined signal capturing and 

projection unit illustrated in Figure 6A. 

Figure 7A shows schematically interactive spectacles in accordance with a 
sixth embodiment. 

Figure 7B is a detailed drawing of a combined signal capturing and 
10 projection unit illustrated in Figure 7A. 

Figure 8 shows schematically interactive spectacles in accordance with a 
seventh embodiment. 

Figure 9 shows schematically interactive spectacles in accordance with an 
eighth embodiment. 

15 Figure 10A shows schematically interactive spectacles in accordance with 

a ninth embodiment. 

Figure 108 is a front view of spectacles in accordance with a ninth 
embodiment. 

Figure 1 1 A illustrates the naturally perceived field of view of a user of an 
20 information system designed in accordance with a tenth embodiment. 

Figure 1 1 B illustrates the naturally perceived field of view of a user of an 
information system designed in accordance with a tenth embodiment. 
Figure 11C is a schematic representation of a scan pattern. 
Figure 11D is a schematic representation of a modified scan pattern. 
25 Figure 12A illustrates the naturally perceived field of view of a user of an 

information system designed in accordance with an eleventh embodiment. 

Figure 12B illustrates the naturally perceived field of view of a user of an 
information system designed in accordance with an eleventh embodiment. 

Figure 12C illustrates the naturally perceived field of view of a user of an 
30 information system designed in accordance with an eleventh embodiment. 
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Figure 12D illustrates the naturally perceived field of view of a user of an 
information system designed in accordance with an eleventh embodiment. 

Figure 12E illustrates the naturally perceived field of view of a user of an 
information system designed in accordance with an eleventh embodiment. 
5 Figure 13A is the naturally perceived field of view of a user of an 

information system designed in accordance with a twelfth embodiment. 

Figure 1 3B is the naturally perceived field of view of a user of an 
information system designed in accordance with a twelfth embodiment. 

Figure 14A shows schematically an information system in accordance with 
10 the invention in accordance with a thirteenth embodiment. 

Figure 14B shows schematically an information system in accordance with 
the invention in accordance with a thirteenth embodiment. 

Figure 15 is an information system in accordance with the invention in 
accordance with a fourteenth embodiment. 
15 Figure 16 is a schematic representation of an information system in 

accordance with the invention in accordance with a fifteenth embodiment. 

Figure 17 is an optical system in accordance with a sixteenth embodiment. 

In the description of the figures, similar or identical objects are designated 
with similar or identically ending reference signs. Many of the illustrated objects 
20 comprise symmetrical or complementary components that are distinguished via a 
supplementary letter, e.g. "L" for left and "R" for right, after the reference sign. If 
the statement applies to each individual component of such a symmetrical or 
complementary grouping, the supplementary letter is left out in some cases for 
the sake of clarity. 

25 

Detailed Description 

Figure 1 illustrates schematically an interactive data view and command 
system 100 in a broadest sense of the word, as an information system. The 
information system 100 is embodied in the form of an interactive spectacle 
30 system 120, /.e. interactive spectacles 120, which comprise two optical 
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apparatuses 150. Preferably, the optical apparatuses 150 are respectively 
located in an inner side of a left 121 L or right 121 R temple of the spectacles 120. 
Depending on the field of application, other arrangements of the optical 
apparatuses that do not disturb the view, e.g. in the region of a bridge 122 of the 
5 spectacles 1 20 that crosses the root of the nose of a user are also appropriate. 

The optical apparatus is connected to a processor unit 140 via connection 
lines 101. If the optical apparatuses comprise photodetectors and/or light 
sources, the connection lines serve the transmission of electrical detection and/or 
control signals. The photodetectors and/or light sources can, however, be 

10 located in the processor unit 140 and be connected to the optical apparatuses 
150 of the spectacles 120 via light-conducting connection lines 101. This 
contributes to reducing the weight of the spectacles 120. 

Still referring to Figure 1, a communication interface 196 (indicated by 
dash-dots) to which con-esponding output signals from the signal processing unit 

15 140 are transmitted in order to establish a communicative connection with an 
external information unit 198 via a preferably mobile communication unit 197 that 
is connected to the interface 196 via a signal line 196a. In the most general 
case, the extemal infonnation unit 198 is a database or data files that can be 
accessed via suitable protocols and/or over the Internet. 

20 A portable telephone, a laptop or a palmtop can, for example, serve as a 

mobile communication unit 197, wherein the connection for the remote data 
transmission can be carried out on the basis of all typical protocols such as GSM, 
UMTS, CDMA, TDMA or DECT. For example, a TDMA communication method 
and apparatus is described in U.S. Patent 6,160,800, which is incorporated by 

25 reference. A CDMA communication method and apparatus is described in U.S. 
Patent 5,878,036, which is incorporated by reference. Another mobile 
telecomunication system is described in PCT Application PCT/SE99/02013 
(published as WO 00/28769, which is incorporated by reference. A technique for 
providing a secure link in a mobile communication system is described in U.S. 

30 Patent 6,301 ,479, which is also incorporated by reference. 
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The signal line 196a is bi-directional so that corresponding request signals 
can be returned via the signal line, as an interactive data exchange between the 
communication unit 197 and the database 198, to the interface 196 and, from 
there, to the signal processing unit 140. A control apparatus, not shown in detail 
and preferably integrated into the processor unit 140, ensures that the request 
signals are converted into the desired operation signals, with which either the 
optical unit 150 is Instigated to place further image information onto the retina 
and/or at least a further information reproduction device, such as e.g. a 
headphone unit 150a, is instigated to transmit additional information to the 
operator. When a further information reproduction system 150a Is provided, a 
further signal line 101 preferably exists that is lead to the processor unit 140. 

A microphone 150a is indicated by a dash-dotted line that is a component 
of the data view and command system and via which speech input can be carried 
out. A corresponding signal line 101b is lead from the spectacle system to the 
processor unit. Speech-controlled operation of the interactive data view and 
command system can be carried out via the microphone. For example, control 
operations such as paging through display menus can be executed or particular 
system actions such as e.g. the triggering of select or deselect operations can be 
triggered. 

The information system Is capable of processing data from the optical unit 
and the communication unit on the fly and simultaneously and, as the case may 
be, as a function of additional control commands from the operator that are either 
supplied to the processing unit via the line 101 or the line 101b, control the signal 
processing set-up 140 via the communication interface () and/or the further 
Information reproduction unit 150a such that the desired result of the interactive 
communication is obtainable. It Is thus, for Instance, possible to convey acoustic 
or Image infomiatlon to the operator that has been downloaded from the Internet 
via the communication unit upon gaze or verbal command. In the case of optical 
information, said is superimposed, precisely positioned, onto Image falling onto 
the eye. 



- 24 - 



A further possibility of controlling the data view and signal processing set- 
up - implemented via the components 140, 196, 197 - consists e.g. of placing a 
control menu onto the operator's retina via the optical unit 150. Since the image 
on the retina is captured cyclically, preferably with a sampling frequency of 100 
Hz, the optical unit 150, in conjunction with the processor unit 140, can 
determine, at any point in time, which image is situated at the center of the field 
of vision since this image is held in the fovea centralis. When the operator thus 
focuses on a particular menu item or a biended-in program control symbol and 
when, at this point in time, a particular trigger signal is generated, the 
corresponding operation is called. The trigger signal can be generated by a 
suitable button provided on the system, via the speech input system (microphone 
150b in connection with a speech decoder in the processor unit) or else in an 
optical manner, e.g. by using a blink of an eye consciously executed at that 
moment as a trigger. 

Naturally, modifications of the system are conceivable without leaving the 
spirit of the invention. Consequently, the mobile data transmission set-up 197 
can naturally be combined with the interface 196 and the signal-processing set- 
up 140 to a unit, as indicated by the dash-dotted line Z. 

The data transmission set-up can comprise a preferably portable computer 
such as e.g. a laptop or palmtop that is equipped with a suitable remote data 
transmission interface. An electromechanical control unit such as e.g. a control 
mouse can also be provided as a control unit for activating particular operations 
of the system. 

Since the optical apparatus constantly samples the operator's retina, it can 
capture the retina's structure - particularly when a suitable wavelength band, for 
instance the infrared band, is chosen for the scan beam - so that unique 
associable user / wearer ID data relating to the operator can be intermediately 
stored as a data record in the signal processing unit 140. The data record can 
then be employed for wearer identification / for personalized set-up of the system 
to the respective operator. 
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The system is suited, in a particularly advantageous manner, for use in the 
medical field, in particular in the field of ophthalmology, as a therapeutic or 
analytic device, as well as in military applications relating to accessing large 
amounts of data in a mobile environment and to using corresponding data 
5 downloaded from the database for the operator as contemporaneously as 
possible. 

In the following, various embodiments of the optical apparatus are 
described that can be used in an advantageous manner in combination with the 
data view and command system in accordance with the invention. Furthermore, 

10 information systems are described that contain various, equally applicable, 
modifications of the optical unit and that are sensibly combinable with the data 
view and command system. For better understanding of the manner of operation 
of the optical apparatus and the system combined therewith, however, attention 
will first be given to Figure 2 and, based on this representation, the structure of 

15 the human eye will be elucidated. 

To fully understand the information system 100, Figure 2 provides a 
detailed view of an eye 280 in cross-section. The eye 280, which is situated in 
two eye sockets 20 (lat. orbita) formed of skull bone in the head of a person and 
which is to be understood here in the sense of an eyeball 280, consists of a 

20 chamber surrounded by a translucent cornea 283 and a visibly white sclera 28. 
The sclera 28 is covered on its side facing the inside of the eye 280 by a choroid 
coat 287 that supports, also on its inner side, a light-sensitive retina 281 and 
supplies same with blood. Due to its pigmentation, the choroid coat 287 hinders 
a scattering of the light falling thereon that could disturb visual acuity. 

25 The tissue of the retina 281 comprises two types of photoreceptor cells, 

/.e. rods and cones (not shown), which provide the person with the ability to see. 
These photoreceptor cells absorb the light focuses by an eye lens 282 in a range 
of wavelengths from roughly 380-760 nm and convert it, via a chain of chemical 
reactions, into electric nerve signals. The signals of the various nerve cells of the 

30 retina 281 are then passed on to the brain via an optic nerve 25 and processed 
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there to a perceptible image. The numerous, highly light-receptive rods, roughly 
120 million in number, are specialized for signal detection in twilight (so-called 
scotopic vision) and yield a grey-scale image. The roughly 6.5 million, 
comparatively less light-receptive cones, in comparison, are responsible for color 
5 vision during daylight (so-called photopic vision). During light absorption, an 
oxidation of pigments in the photoreceptor cells takes place. For the 
regeneration of the pigments, the cones require roughly six minutes and the rods 
require roughly 30 minutes. An observation period of roughly 200 milliseconds is 
necessary until the visual stimulus via the photoreceptors sets in and a reception 

10 of information via the retina 281 takes place. 

The retina 281 comprises a depression 286 that appears somewhat more 
strongly pigmented due to its higher density of cones in comparison with the rest 
of retina. This depression 286, which is typically called the fovea centralis, lies in 
a region of the retina known as the macula and represents the region of keenest 

15 vision. The fovea centralis 286 is only occupied by cones, comprises a very high 
cone density and encompasses solely roughly 0.01% of the retina surface. The 
optic nerve 25 enters into the inside of the eye via a sieve-like opening in the 
sclera 28 in an area vis-a-vis the lens 282 designated with the reference sign 
288. This area 288 does not comprise any photoreceptor cells, whence it is 

20 named "blind spot." 

The chamber formed by the cornea 283 and the sclera 28 is partitioned by 
a deformable lens 282 and muscular ciliary processes 23 that supports the lens 
282. The portion of the chamber lying the lens 282 and the retina 281 , which 
makes up roughly two-thirds of the eyeball, forms a so-called vitreous humor 21, 

25 a gelatinous structure that consists, to over 98%, of water and that supports and 
protects the retina 281. The portion of the chamber lying between the cornea 
283 and the lens 282 carries the name anterior chamber 22 and contains a fluid 
that nourishes the cornea 283. In its neutral shape, the lens 282 typically refracts 
the light falling on the eye such the far-away field of view is sharply imaged onto 

30 the retina 281 . Through contraction/relaxation of the muscles of the ciliary 
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processes 23, the shape and thus also the refractive characteristics of the lens 
282 can be varied over a wide range in order to allow e.g. a sharp imaging of 
close-lying objects in the field of view onto the retina 281 . In most cases, the 
person affected is unaware of this process. 
5 An aperture 285 of variable diameter and consisting of colored tissue 

regulates the light falling onto the light-sensitive portions of the eye 280 and 
gives the eye 280 its characteristic coloring. Thus aperture 285 is located in the 
anterior chamber 22 directly in front of the lens 282 and is called the iris 285. 
Due to the low amount of light backscattered by the lens 282, the vitreous humor 
10 21 and the retina 281, the central region of the iris 285 appears black and called 

H the pupil 284. The regulation of the pupil size is also carried out by the person 

p subconsciously. 

S^J The eye 280 is connected to the skull via six muscles 24 that run partially 

W parallel and partially oblique to one another that allow a pivoting of the eye 280 

15 and subsequently a change of the gaze direction. The binocular field of view 
captured without moving the eyes 280 encompasses roughly 170° horizontally 
and roughly 110° vertically. If the eyes 280 are moved, a binocular field of vision 
5 of roughly IdO"" horizontally and roughly 190° vertically can be captured. The 

O region of keenest vision captured by the fovea centralis 286 encompasses solely 

20 roughly r. A fictitious axis through the center of this region is called the visual 
axis and conresponds to the direction of gaze. A rotation of the eye around the 
visual / optical axis is also enabled via the muscles 24. 

The six muscles 24 are responsible for all movements of the eye. During 
observation of a fixed point, so-called micro-tremors of the eye 280 take place, 
25 during which the eyes 280 tremble lightly in order to avoid a temporary 

exhaustion of the ability of the affected photoreceptor cells to chemically react to 
a persistent stimulus. During a change of the direction of gaze or a movement of 
the head, so-called saccadic movements take place, with whose aid the fovea 
centralis 286 is directed to its new target of fixation or held on its previous target 
30 of fixation. During these highly complexly structured movements, the eye 280 is 
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Involuntarily moved back and forth at a small amplitude of up to several tens of 
degrees and with an extremely fast angular velocity of up to several hundred 
degrees per second. During the tracking of a moving object, the eye 280 
achieves angular velocities of only one to two hundred degrees per second. 
5 To protect the eyeball 280, people have a movable fold of skin, i.e. an 

upper lid 27a and a lower lid 27b that allow a closing of the eye socket 20 against 
external Influences. The lids 27a and 27b close as a reflex in the presence of 
foreign objects or strong light. Moreover, the lids 27a and 27b provide, via 
regular, typically involuntary blinking, for an evenly distributed film of tears on the 

10 cornea 283 that washes the outer surface of the cornea 283 and protects it from 
drying out. The lids 27a and 27b also comprise lashes 27c that also protect the 
eye 280 from dust. A conjunctiva 26 covers the space between the lids 27a,27b, 
the eye socket 20 and the eyeball 280. The conjunctiva 26 merges, on the one 
hand, with the inner side of the lid and, on the other hand, with the cornea 283 

15 and represents a second wall of protection against the penetration of gernis and 
foreign objects. 

Figure 3 shows schematically a first embodiment of the optical apparatus 
of the, as described above, interactive spectacles system / spectacles 320 as a 
component of the interactive data view and command system. A signal capturing 

20 apparatus in the form of a scanning eye scanning apparatus 350D is provided. 
The left half of Figure 3 represents a plan view onto the head of a user 302 
together with spectacles 320 having a right temple 321 R, whereas the right half 
of Figure 3 reflects a cross-section of the spectacles 320 running through the left 
temple 321 L. No further components of the information system 100 are shown in 

25 Figure 3 other than the apparatuses belonging to the interactive spectacles 320. 

In accordance with the illustrated embodiment, light beams 333a and 333b 
falling onto the eye 380 that originate e.g. from the field of view are sharply 
imaged onto the retina 381 by the lens 382 as a correlated image and are 
reflected back by the retina 381 as a retinal reflex image. A light beam 331 that 

30 has been reflected back in this manner passes again, in the opposite direction, 
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through the lens 382, is focused via two concave mirrors 322 and 323 that 
belong to the mirror system of the spectacles 320 and is directed, as shown, onto 
a scanning eye scanning apparatus 350D. The eye scanning apparatus 350D 
comprises a signal capturing apparatus 351 in the form of a photodetector 361 
that captures the light beam 331 reflected back from the retina 381 as well as two 
movable flat mirrors 352H and 353V that effect a horizontal / vertical deflection of 
the light beam 331 onto the photodetector 351 . In accordance with the 
embodiment of Figure 3, the spectacles 320 additionally comprises a light trap 
324 that prohibits an incidence of light from undesired directions of incidence. To 
simplify the mirror system of the spectacles 320, this mirror 323 can be 
implemented via a mirrored inner surface of the spectacle lens. However, the 
surface must have a particular shape in order to facilitate a capturing of the entire 
retinal reflex image even when the eye 380 is possibly in a skewed position. 
This, on the other hand, limits the design freedoms of the spectacles 320. 

A serial, point-by-point scanning of the retinal reflex image as a pixel 
sequence is carried out via the combination of a point-shaped detector 351 and 
corresponding control of the flat mirrors 352H and 352V. Preferably, the retina 
381 is scanned with a circular, spiral or elliptical scan pattem as described in 
the PCT Application PCT/EP97/041 88 (published as WO98/05992) and U.S. 
Application 09/462,440. This has the advantage that the flat mirrors 352 can be 
driven without jerky movements and that a higher pixel density (number of pixels 
per unit of area of the retina) can be captured in the region of the fovea centralis 
286. 

Preferably, a suitable synchronization operation for determining the 
cun-ent optical axis is carried out prior to the capturing operation - to the respect 
that it has not already been earned out in a previous projection operation - so that 
the scan operation can be carried out centered to the eye. 

Figure 4 shows schematically an embodiment of the interactive spectacles 
420 in which an output apparatus is in the form of a scanning projection 
apparatus 450P. The left half of Figure 4 represents a plan view onto the head of 
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a user 402 together with spectacles 420 having a right temple 421 R, whereas the 
right half of Figure 4 reflects a cross-section of the spectacles 420 running 
through the left temple 421 L. No further components of the information system 
100 are shown in Figure 4 other than the apparatuses belonging to the 
5 interactive spectacles 420. 

In accordance with the illustrated embodiment, a scanning projection 
apparatus 450P comprises a light source 453, e.g. a laser diode or an LED 
focused via a lens system that emits a projection light beam 432 as well as two 
movable flat mirrors 454H and 454V. The projection light beam 432 is directed 

10 via the movable flat mirrors 454H and 454V onto a mirror system of the 

spectacles 420 that comprises two concave mirrors 422 and 423 that projects the 
projection light beam 432 onto the lens 482 of an eye 480 and, In the end. onto 
the retina 481 . To simplify the min-or system of the spectacles 420, this mirror 
423 can be implemented via a mirrored inner surface of the spectacle lens. 

15 However, the surface must have a particular shape in order to facilitate a 

capturing of the entire retinal reflex image even when the eye 480 is possibly in a 
skewed position. This, on the other hand, limits the design freedoms of the 
spectacles 420. To avoid the incidence of light that would be disturbing, the 
spectacles 420 can be equipped with a light trap 424 that hinders the incidence 

20 of from undesired direction of incidence. 

A serial, point-for-point projection of an image is carried out via the 
combination of a point-shaped light source 553 with corresponding control of the 
flat mirrors 452H and 452V that respectively effect a horizontal / vertical 
deflection of the projection light beam 432. The projection is preferably carried 

25 out. as described in US Patent 6,227.667 and US Application 09/462,440, in a 
circular, spiral or elliptical scan pattern. This has the advantage that the flat 
mirrors 452 can be driven without jerky movements and that a higher pixel 
density can be projected onto the retina 481 in the region of the fovea centralis 
286. 
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The degree of perception of an image projected into the eye 480 can be 
controlled in relation to the naturally perceived image via the brightness of the 
projected pixels. However, retinal perception is a highly complex process in 
which psychological effects also play a strong role. In this respect, reference is 
5 made to the relevant literature of the field. 

In simplified form, however, one can say that the retina 481 adapts to the 
brightness of the total light falling thereon. It is known, for example, that the 
slight glow of the clock of a radio alarm that cannot even be perceived in daylight 
can appear to illuminate an entire room in a dark night. On the other hand, the 

10 strong light of headlights of approaching vehicles is barely perceptible In daylight. 
The brightness of a single pixel is thus perceived in relation to the pixels 
othenwise perceived. The retina 481 functions similarly when observed locally. If 
the brightness of a pixel projected onto a region of the retina 481 exceeds the 
brightness of the light otherwise falling onto this region by roughly 10%, then 

15 solely the projected pixel is effectively perceived by this region of the retina 481 
in lieu of the other light. Due to psychological effects, the exact value can also lie 
between 5%-10%, 10%-15% or even 15%-20% instead of at 10%. 

Preferably, a suitable synchronization operation for determining the 
current optical axis is carried out prior to the projection operation - to the respect 

20 that it has not already been canried out in a previous scanning operation - so that 
the projection operation can be carried out centered to the eye. 

Figure 5A shows schematically interactive spectacles 520 in accordance 
with a fourth preferred embodiment in which a combined signal capture and 
projection apparatus 550 is attached to the spectacles 520 in the region of the 

25 bridge 522. Referring to Figure 5B, the combined signal capture and projection 
apparatus 550 comprises both a projection apparatus 553 as well as a signal 
capturing apparatus that are housed together in a protective housing 558. Light 
beams 530 make their way into the inside of the housing 558 and vice-versa via 
a translucent window 559 in an outer wall of the housing 558. The sealing of the 

30 housing 558 via the window 559, however, keeps dust, sweat and other foreign 
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materials from disturbing operation of the combined signal capture and projection 
apparatus 550. 

Light beams 530, 530a, 530b are captured / projected analogously to the 
described systems in accordance with Figures 3 and 4. The interactive 
5 spectacles 520 can be simplified, however, in their construction by replacing the 
min-ors 352/452, which are separate in the prior art, for vertical / horizontal 
deflection of the respective light beams 331/432 with a swiveling mirror 552 / 
554. For the purpose of achieving a compact design, a partially transmissive 
mirror 556 can serve to allow separate beam paths within the housing 558 for the 

10 light 530 falling or projected through the window 559. Preferably, the inner side 
of the spectacle lens is provided with a surface 523 that strongly reflects beams 
incident from this direction that is used as a mirror for the beam path between the 
eye 580 and the combined signal capturing and projection apparatus 550. This 
contributes to a reduction of the components required and results, in the 

15 illustrated embodiment, in a simplified transmission-efficient beam path 530 in 
which the light beam 530 between the eye 580 and the projection / signal 
capturing apparatus 553 / 551 is only reflected three times. As described above, 
however, this results in a limitation of the design freedoms of the spectacles 520. 
The freedom of movement necessary for a swiveling motion of the mirror 

20 552, 554 can be achieved, for example, via a Cardan joint or spring suspension 
of the mirror 552, 554. Possible embodiments of such a swiveling mirror are 
known to the person skilled in the art, e.g. from the field of microtechnology. 
Further solutions to the present deflection problem in which the respective light 
beam 530 is deflected on the basis of electrochrome, holographic, electro- 

25 holographic or other light refraction or light reflection mechanisms are easily 
conceivable and equally applicable. 

Although the interactive spectacles 520 are shown in a minimalist 
embodiment in which a combined signal capturing and projection apparatus 550 
is solely provided for the left eye 580, it is self-evident that a second combined 

30 signal capturing and projection apparatus 550 having a mirror image design can 
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be provided for the non-illustrated right eye, if necessary, in the region of the right 
half of the bridge 522. 

Figure 6A shows schematically a modification of the spectacles 520 
illustrated in Figures 5A and 5B. Interactive spectacles 620 in accordance with a 
5 fifth preferred embodiment utilize the left combined signal capturing and 

projection apparatus 550L situated in the region lying between the left spectacle 
lens 624L and the left temple 621 L and the right combined signal capturing and 
projection apparatus 650R situated in the region lying between the right 
spectacle lens 624R and the left temple 621 R. 
10 Such an arrangement of the combined signal capturing and projection 

S apparatuses 650L, 650R vis-a-vis the respective spectacle lenses 624L, 624R 

O and the respective eyes 680 is typically associated with the necessity of either 

ffl 

§1 providing several mirrors along the beam path 630 (cf. mirror 322 and 323 in 

Figure 3) or bestowing the respective spectacle lens 624L, 624R with a particular 
ry 15 form in order to guarantee a capture of all regions of the retina 681 . This, 
g however, significantly limits the design freedoms of the spectacles 620. In order 

to circumvent this problem, the interactive spectacles 620 in accordance with 
□ Figure 6 propose spectacle lens 624L, 624R whose inner sides are provided with 

^ a respective holographic coating 623L, 623R. Such a holographic coating 623 is 

20 capable of emulating an arbitrary reflection topology. For example, a 

holographically coated, flat surface can act like a spherically curved surface. 
Similarly, a holographically coated, spherically curved surface can act like a flat 
surface. The change of the effective reflection topology depends solely on the 
holographic content of the coating. In accordance with the Figure, the 
25 holographic coating 623L and 623R are designed and situated as mirror images 
to one another. 

Figure 6B shows schematically the combined signal capturing and 
projection apparatus 650L. Analogously to the combined signal capturing and 
projection apparatus 550, illustrated in Figure 5B, apparatus 650L comprises a 
30 housing 658, a projection apparatus 653 and a signal capturing apparatus 651 , 
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respective swiveling mirrors 652 and 654, a partially transmissive mirror 656 and 
a housing window 659. 

Figure 7A shows schematically a modification of the spectacles 520 
shown in Figures 5A and 5B. Interactive spectacles 720 in accordance with a 
5 sixth, preferred embodiment utilize the left combined signal capturing and 

projection apparatus 750L situated in the region lying between the left spectacle 
lens 724L and the left temple 721 L and the right combined signal capturing and 
projection apparatus 750R situated in the region lying between the right 
spectacle lens 724R and the left temple 721 R. 

10 Figure 7B shows schematically the combined signal capturing and 

projection apparatus 750L Analogously to the combined signal capturing and 
projection apparatus 550 illustrated in Figure 5B, it comprises a housing 758, a 
projection apparatus 753 and a signal capturing apparatus 751, respective 
swiveling mirrors 752 and 754, a partially transmissive mirror 756 and a housing 

15 window 759. 

The problem of the beam path 730 touched upon above is solved in this 
embodiment in a space-saving manner via a special design of pads 725L and 
725R. Typically, spectacles 720 are supported on the root of the nose either 
through the bridge 722 or through so-called pads 725. In their typically 

20 commercial design, pads are relatively flat, slightly curved and oval. Moreover, 
they are either hingably or swivably mounted on a projection extending from the 
bridge 722 in order to ensure comfortable contact of the pads to the side 
surfaces of the root of the nose. In the illustrated embodiment, the pads 725 are 
formed as fixed-shaped, elongated units that project from the spectacles 720 in 

25 the direction of eye 780 in the region of the bridge 722. On their respective 
longitudinal sides facing the nose, the pads 725 form the contact surfaces that 
support themselves on the root of the nose. In their end region lying across from 
the spectacles 720, the pads 725 comprise a support surface on the respective 
side facing the eye that is provided with a mirror or a mirroring coating, e.g. a 

30 metallic coating or a holographic coating. 
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Although the frame of the spectacles 720, including the pads 725, has a 
principally fixed form, both quasi-static, e.g. due to material fatigue and/or 
temperature changes, as well as dynamic deformations of the frame. Particularly 
when the spectacles 720 are put on and during activities in which vibrations are 
5 commonplaces, changes to the relative arrangement of the respective spectacle 
components to one another result. The relative location of the spectacles 720 
vis-a-vis the eye 780 is also not constant. Accordingly, both the optical system of 
the spectacles 720, Le, those system components that contribute to the optical 
signal capturing and/or the optical projection as well as any processing system 

10 connected to thereto, must be conceived and designed such that such changes 
in the arrangement can be taken into consideration and/or compensated and/or 
do not cause any extraordinary operational disturbances. This also holds for all 
types of interactive spectacle systems. 

The problem addressed above can be overcome in particular through a 

15 suitable signal processing of the captured signals and the signals to be 
generated. Furthermore, an optical marker fixed disposed on the spectacle 
frame in the vicinity of the typical beam path 730 can be additionally detected on 
a regular basis or on demand via the signal capturing apparatus 751 for the 
purpose of calibrating its optical system. 

20 Figure 8 shows schematically a modification of the spectacles 520 

illustrated in Figures 5A and 5B, interactive spectacles in accordance with a 
seventh preferred embodiment in which the signal capturing apparatus 851 of the 
combined signal capturing and projection apparatus 850 is capable of at least 
partially capturing the corneal reflex image. 

25 The comea is typically formed rotationally symmetric to the optical axis. 

Beams that fall perpendicularly onto a central region of the cornea are thus 
confocal to the optical system of the eye 880 and form the basis of the image 
truly perceived on the retina 881. Moreover, the cornea 883 consists, to a large 
degree, of water and exhibits, for this reason, a very high degree of reflectivity at 

30 a wavelength of roughly 1.1 [im. Since this wavelength lies in the infrared 



- 36 - 



spectral region, a capturing of the corneal reflex image is primarily suitable for 
infrared applications, e.g. for night vision devices. Reflections occur not only on 
the outer, concave corneal surface, however, but also on the inside of the 
cornea. Moreover, due to its structure, the cornea 883 does not effect mirror-like 
5 reflections but instead effects a diffuse reflection that becomes more diffuse with 
increasing depth of the act of reflection within the cornea. 

In order to obtain a meaningful corneal reflex image, effectively only those 
beams that fall perpendicularly onto a central region of the cornea are captured 
in the illustrated embodiment. This is achieved through several measures. 

10 Firstly, the spectacle lens 824 situated in front of the eye whose side facing the 
eye 880 is provided with a surface 823 that is high reflective for beams incident 
from this direction comprises a specially designed shape that focuses the light 
perpendicularly reflected from the cornea such that it falls onto the signal 
capturing apparatus 851 as light beams 834 that run nearly in parallel whereas 

15 light that is reflected non-perpendicularly from the cornea is deflected in another 
direction. Alternatively, the spectacle lens 824 can be designed in another 
fashion, yet comprise a partially transmissive, holographically reflecting layer 823 
that likewise effects such a focusing of the light reflected perpendicularly from the 
cornea such that it falls onto the signal capturing apparatus 851 as light beams 

20 834 that run nearly in parallel, whereas light non-perpendicularly reflected by the 
cornea is deflected in another direction. Secondly, an aperture 857 is provided 
shortly in front of the signal capturing apparatus 851 that prohibits a capturing of 
those light beams whose incident angle lies outside a narrow range of incident 
angles of the light beams 834 that run nearly in parallel as described above. 

25 Figure 8 shows schematically a modification of the spectacles 520 

illustrated in Figures 5A and 5B. The interactive spectacles in accordance with 
another preferred embodiment utilize the signal capturing apparatus 851 of the 
combined signal capturing and projection apparatus 850 capable of at least 
partially capturing the corneal reflex image. 
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The cornea is typically formed rotationally symmetric to the optical axis. 
Beams that fall perpendicularly onto a central region of the cornea are thus 
confocal to the optical system of the eye 880 and form the basis of the image 
truly perceived on the retina 881. Moreover, the cornea 883 consists, to a large 
5 degree, of water and exhibits, for this reason, a very high degree of reflectivity at 
a wavelength of roughly 1.1 |jm. Since this wavelength lies in the infrared 
spectral region, a capturing of the corneal reflex image is primarily suitable for 
infrared applications, e.g. for night vision devices. Reflections occur not only on 
the outer, concave corneal surface, however, but also on the inside of the 

10 cornea. Moreover, due to its structure, the cornea 883 does not effect mirror-like 
reflections but instead effects a diffuse reflection that becomes more diffuse with 
increasing depth of the act of reflection within the cornea. 

In order to obtain a meaningful corneal reflex image, effectively only those 
beams that fall perpendicularly onto a central region of the cornea are captured 

15 in the illustrated embodiment. This is achieved through several measures. 

Firstly, the spectacle lens 824 situated in front of the eye whose side facing the 
eye 880 is provided with a surface 823 that is high reflective for beams incident 
from this direction comprises a specially designed shape that focuses the light 
perpendicularly reflected from the cornea such that it falls onto the signal 

20 capturing apparatus 851 as light beams 834 that run nearly in parallel whereas 
light that is reflected non-perpendicularly from the cornea is deflected in another 
direction. Alternatively, the spectacle lens 824 can be designed in another 
fashion, yet comprise a partially transmissive, holographically reflecting layer 823 
that likewise effects such a focusing of the light reflected perpendicularly from the 

25 cornea such that it falls onto the signal capturing apparatus 851 as light beams 
834 that run nearly in parallel, whereas light non-perpendicularly reflected by the 
cornea is deflected in another direction. Secondly, an aperture 857 is provided 
shortly in front of the signal capturing apparatus 851 that prohibits a capturing of 
those light beams whose incident angle lies outside a narrow range of incident 

30 angles of the light beams 834 that run nearly in parallel as described above. 
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Figure 9 shows schematically a modification of the spectacles 520 
illustrated in Figures 5A and 5B. The interactive spectacles in accordance with 
another preferred embodiment utilize a spherical or spherical-acting, partially 
transmissive, mirroring supplementary element 929 arranged between the 
5 spectacle lens 924 and the eye 980. Preferably, the supplementary element 929 
is arranged confocal to the optical system of the eye 980. 

The degree of reflectivity of the supplementary element 929 can be 
adapted to the requirements of the information system. One can choose 
between a high degree of reflectivity, which allows very good capturing of light 

10 beams 933a-933c directed onto the eye 980, and a low degree of reflectivity, 
which avoids impairment of the perception carried out by the eye 980. 
Preferably, the supplementary element 929 exhibits a low {e.g. less than 10%), 
homogenous degree of reflectivity over Its entire reflective surface. On the other 
hand, reflecting organs of the eye 980, for instance the cornea 983 or the retina 

15 981 , exhibit, in part, very strong local reflective dependencies. Similar 

statements hold for the spectral reflective dependencies of the supplementary 
element and/or the reflecting organs of the eye 980. Whereas the supplementary 
element 929 can be preferably designed such that it exhibits a homogeneous 
degree of reflectivity over all relevant spectral ranges, the various organs of the 

20 eye 980 exhibit highly differing degrees of absorption that, in many cases, are 
also subject to strong local variations. 

Excepting partial reflection, the supplementary element 929 should have 
as little effect as possible on the light falling thereon. For this reason, the 
supplementary element 929 is preferably manufactured of a homogenous, 

25 translucent and uncolored material and is manufactured to have a constant 

thickness in the direction of the light beams directed toward the center of the eye. 
By applying an anti-reflective coating to the side of the supplementary element 
929 facing the eye 980, improved translucency can be achieved. 

The reflecting contour of such a supplementary element 929 is well 

30 defined and can thus be supplied to the information system as known 
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information, whereas the contour of the relevant reflecting organs of the eye 980 
nnust first be determined. In some respects, latter encompasses significant 
difficulties. The capturing of the light beams 933a-933c directed onto the eye 
980 can thus yield valuable images of the field of vision. 
5 In the illustrated embodiment, effectively only those beams that fall 

perpendicularly onto the supplementary element 929 are captured. This is 
achieved through the following measures: 

Due to the partially reflective surface of the supplementary element 929, a 
corresponding portion of those beams 933a-933c that fall perpendicularly onto 

10 the surface of the supplementary element 929 are reflected back perpendicularly, 
whereas other beams are reflected back from the surface of the supplementary 
element 929 correspondingly skewed in accordance with the law of reflection 
"The angle of incidence equals the angle of reflection." The light beams reflected 
back perpendicular to the surface of the supplementary element 929 travel back 

15 the same way they came and are thus incident upon the spectacle lens 924 

situated in front of the eye. The side of the spectacle lens 924 facing the eye 980 
is provided with a surface 923 that is highly reflective for beams incident from this 
direction and comprises a specially designed shape or a specially formed coating 
that focuses the light beams reflected perpendicularly by the supplementary 

20 element such that they fall onto the signal capturing apparatus 951 as light 
beams 934 that run nearly in parallel whereas light beams reflected non- 
perpendicularly by the supplementary element are deflected in another direction. 
In addition, an aperture 957 is provided shortly in front of the signal capturing 
apparatus 951 that prohibits a capturing of those light beams whose incident 

25 angle lies outside a narrow range of incident angles of the light beams 934 that 
run nearly in parallel as described above. 

If the image of the field of view captured via the supplementary element 
929 is to be the basis for a projection correlated with the actually perceived field 
of view, then the correlation between the captured light and the perceived field of 

30 view must be determined. In accordance with the embodiment of Figure 9, this 
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correlation is achieved through a preferably confocal arrangement of the 
supplementary element 929 to the optical system of the eye 980. Thus, 
preferably, the supplementary element 929 is fastened to the spectacles via an 
adjustable suspension such that the position of the supplementary element 929 
5 can be adjusted in both vertical as well as in two horizontal directions. 

To obtain confocal arrangement, the supplementary element 929 is 
situated rotationally symmetric to the optical axis and is spaced from the lens 982 
such that the optical mid-point of the optical system of the eye agrees with the 
mid-point of the sphere defined by the spherical or spherical-acting 
10 supplementary element The optical axis can be sufficiently determined for this 
purpose via the orientation of the pupil 984 that is easily recognizable via its 
y sharp contours and whose orientation is easily detenninable due to its round 

y 

y i shape. In addition, due to the spherical or spherical-acting shape of the 

pi supplementary element 929, no pivoting of the supplementary element 929 

C 15 around the possible pivotal axes of the eye 980 is necessary to ensure 
i confocality since, even the case of a skewing of the eye, at least a substantial 

^ portion of the supplementary element 929 remains, in tenns of optics, rotationally 

y symmetric to the optical axis through a corresponding vertical and/or horizontal 

P shift of the supplementary element 929. As regards the distance to the lens 982, 

20 there are various possibilities for determining the necessary distance. For 

example, an optical or acoustic measurement of the cornea 983 can be carried 
out whose curvature yields a very good estimate of the correct location of the 
supplementary element 929. Retinal or cornea reflex images can also be at least 
partially captured, and the correct distance can be determined on the basis of a 
25 comparison of the reflex images with the light captured via the supplementary 
element 929. 

Due to the spherical or spherical-acting implementation (e.g. through a 
holographic coating) of the partially reflecting surface of the supplementary 
element 929 as well as through the confocal arrangement of the supplementary 
30 element to the eye 980. solely those beams 933a-933c that fall perpendicularly 
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onto the surface of the supplementary element 929 are confocal to the optical 
system of the eye 980 and thus correspond to the beams falling onto the retina. 

Figure 1 0A is a plan view and Figure 1 0B is a front view of spectacles 
1020 in accordance with another embodiment utilizing two sensor devices 1061R 
5 and 1 061 L. For example, two solid-state cameras (e.g., CCD or TTL cameras) 
are provided to capture additional signal, in particular in the visible field of vision. 
Figure 10B also shows the left eye 1080L and the right eye of a wearer 1 002 
wearing the spectacles 1 020. For the sake of clarity, however, no other features 
of the user 1002 are represented in Figure 10B. 

10 To avoid the occurrence of parallax between the images captured by the 

respective cameras 1061R. 1 061 L and the images received by the eye 
associated therewith, the cameras 1061 should be arranged as coaxially as 
possible to the eyes with regard to their "optical axes." In view of the system size 
of such solid-state cameras 1061 and the current state of the art, it has turned 

15 out to be meaningful to located the cameras 1061 in the front region of the 
respective temples 1021L, 1021R as shown. A mounting in the region of the 
bridge 1022, e.g. in the pads 1025, is also meaningful. After a further 
miniaturization, the solid-state cameras 1061 will be able to be located in the 
spectacle frame over the respective spectacle lenses 1024L, 1024R in order to 

20 achieve further axial identity. It is foreseeable that solid-state and other types of 
light capturing systems will, in the future, be able to be built into the spectacle 
lens 1024, which can naturally be glass, plastic or other translucent material. 
Such an arrangement of the cameras 1061 would allow a signal capturing that is 
coaxial and nearly confocal with the eye 1080L, 1080R. 

25 In a non-coaxial arrangement of the sensor apparatus 1061 to the 

respective eyes 1080L, 1080R, the information obtained from the sensor 
apparatuses 1061 should be brought into correlation with the eyes 1080, as need 
be. Such correlation is particularly important when the sensor apparatuses 1061 
are implemented by cameras 1061 and a superimposed image based on image 
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information obtained from the cameras 1061 is to be projected into the respective 
eyelOSOL, 1080R. 

If the image captured by the cameras 1061 is simply projected on the 
respective eye 1080L, 1080R, so-called parallax occurs, in which the "field of 
5 view" of the respective camera 1 061 L, 1 061 R does not agree with the naturally 
perceived field of view. In particular during a skewing of the eye 1080 deviating 
from the neutral position, or in the case of objects lying closer in the field of view, 
parallax would lead to abnormal perception in the case of superimposition since, 
in such cases, the optical axis of the eye 1080 would lie skewed to the "optical 

10 axis" of the respective camera 1 061 L, 1 061 R. 

During correlation in this sense, only the portion of the image captured by 
the cameras 1061 is projected into the respective eye 1080L, 1080R that lies in 
corresponding "correlation" to the optical axis of the respective eye 1080L, 
1080R. In the simplest case, an at least partial reflex image of the field of view is 

15 captured from the respective 1080L, 1080R via the signal capturing apparatus 
1051. Characteristic pixels that can be found in both the captured reflex image 
as well as in the images captured by the cameras 1061 then serve as reference 
points for a perspective^ correct projection of the image infonnation captured by 
the cameras 1061 onto the eye 1080. Similarly, the signals captured from the 

20 eye 1 080 can serve to determine the gaze direction of the respective eye 1 080L, 
1080R with respect to the coordinate system of the spectacles 1020 in order to 
carry out a mathematically based correlation from this angular information. 

The correlation is also meaningful in the context of system applications in 
which the eyes 1080 are hindered from perceiving the field of view. This is the 

25 case, for example, during use of occluded, so-called "virtual reality" glasses 1020 
(as shown, yet with non-translucent lenses 1024) wherein solely a synthetically 
generated image is presented to the eyes 1080. In such a case, the 
aforementioned correlation can consist, for example, of capturing the gaze 
direction of the eye 1080 as described above and projecting a virtually generated 

30 image that corresponds to the orientation of the respective eye 1080L, 1080R. In 
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this case, however, the spectacles 1020 serve as a coordinate system. If, 
however, the position and orientation of the spectacles 1020 is also determined, 
e.g, on the basis of the images captured by the cameras 1061, then a correlation 
between the respective eye 1080L, 1080R and the surroundings can be created. 
5 Such a system could be used, for example, in a virtual amusement house, similar 
to a house of horrors. Someone who's currently standing on a conveyor belt 
could have, for example, a virtual image projected into the eyes that gives him 
the feeling he is running on floating tree trunks in the middle of a wild river 

We emphasize that the information system described above in connection 

10 with Figures 5 to 10 must not necessarily operate with a combined signal 

capturing and projection apparatus. According to another preferred embodiment, 
the information system may include separate signal capturing and projection 
apparatuses. According to another preferred embodiment, the information 
system may include only a signal capturing apparatus, and according to yet 

15 another preferred embodiment, the information system may include only a signal 
projection apparatus, or in another embodiment none of the two apparatuses. 
Any of the signal capturing or projection apparatuses may execute only partial or 
limited capture or projection. 

Figures 1 1 A and 1 1 B illustrate schematically the use of information system 

20 100 a telescope for a user. Figure 1 1A shows the naturally perceived field of 
view 1 1 90 of a user. Although the field of view 1 1 90 encompasses roughly 170** 
of the surroundings horizontally and roughly 110° of the surroundings vertically, 
solely a small region 1 191 of several degrees around the visual axis forms the 
region of keenest sight 1 191 . 

25 Via its capturing of light from the field of view and the aforementioned 

possibility of a projection of image information into the eye, the information 
system can be designed such that this region 1191 is projected, optically 
enlarged, onto the region of keenest sight 1 191 after corresponding processing 
of the captured pixels via an evaluation apparatus comprised by the information 

30 unit, e.g. upon pressing of a button. As described above, the degree of 



-44- 



perception of an image projected in this manner in relation to the naturally 
perceived image can be controlled via the brightness of the projected pixels. If 
the field-of-view light is captured, for instance, as a reflex image from the eye, a 
spatial or temporal separation of the capturing and the projection will ensure that 

5 the projection does not influence the capturing. 

In the case of a conventional telescope, the spatial relationship to the 
surroundings is lost on account of the fact that the entire field of view is shown in 
enlargement. As a consequence, a person looking through a telescope cannot 
walk or drive at the same time. This phenomena is well known. 

10 Since the information system 100 can determine, by capturing signals 

from the eye, the visual axis / the position of the fovea centralis relative to the 
optical system of the spectacles, the infonnation system is capable of avoiding 
this disadvantage of a conventional telescope. For example, the projection can 
be carried out in a manner shown in Figure 11 B. The system projects a small 

15 region 1 191 lying directly around the visual axis in the natural field of view onto 
the fovea centralis in enlargement, whereas no projected image information is 
superimposed on the remainder of the field of view. The scene peripherally 
perceived by the user thus stays the same in spite of telescopic presentation of 
the most relevant region of the field of view. In order to achieve this effect, the 

20 brightness of the image information projected into the eye must naturally be 

chosen such that the desired relationship of perception between the natural and 
the projected image results. This system also has the advantage that the 
amount of image processing necessary for the enlargement is held within limits 
since only a selected image region 1 191 of the field of view 1 190 is processed. 

25 In accordance with another embodiment, an enlarged image is projected 

into the eye such that the projected image in an annular border region between 
the region of keenest sight 1 191 and the remaining region of the retina is 
enlarged more strongly as it gets closer to the visual axis. In this case, no 
enlargement takes place along the outer edge and along the inner edge an 

30 enlargement takes place with the same "zoom factor" as the enlarged image 
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projected into the inside of the ring, i.e. onto the fovea centralis. Thus, when the 
brightness of the projected image information is correspondingly chosen, a soft 
transition between the peripheral scene and that which Is telescopically seen 
results. 

5 Figures 1 1 C and 1 1 D schematically illustrate the enlargement of the 

image naturally falling onto the fovea centralis by modifying a scan pattern 1 138, 
1 1 39 during the scanning of a reflex image. In Figures 1 1 C and 1 1 D, the 
projection pattern 1137 and scan patterns 1138, 1139 are illustrated in the same 
plane, for the sake of explanation. In general, in the information system 100, the 

10 projection can take place onto the retina, whereas the scanning can takes place, 
for example, from the cornea, and this may be done by different units (or even on 
different eyes of the wearer). 

Figure 1 1C schematically illustrates a typical scan pattern 1 138 that scans 
the region 1 1 89 of the cornea or retina reflecting the field of view. In this vastly 

15 simplified example, it is assumed, for the sake of comprehensibility, that the 
respective pixels of the sequentially scanned image are projected back, after 
image-processing preparation if need be, in their proper sequence as 
corresponding images of the sequential image projected into the eye. In the 
illustrated example, the scan pattern 1 138 thus corresponds to the projection 

20 pattem 1 1 37 in spite of possible spatial or temporal separation of the scan beam 
and the projection beam. If an enlargement of a central region of the field of view 
is desired, then the scanning can be effected in accordance with a modified scan 
pattern 1 139 that effects an increase in the density of the captured pixels in that 
central region. If these pixels captured at higher density are projected back 

25 conrespondingly, yet at lower density during the projection, then an enlarged 
image is the result. 

In accordance with another embodiment, information system 100 is 
constructed and arranged as a guidance system. For this purpose, the 
30 information unit of infomnation system 100 comprises position sensor e.g. 

-46- 



acceleration measurement apparatuses or GPS receivers as well as a data bank 
or data bank connection that supplies orientation data. Such a data bank can be 
implemented e.g. via a CD-ROM carrying the data, a DVD or another 
exchangeable storage medium in connection with a corresponding reading 
5 device. Methods and apparatuses for obtaining position fixing information that, 
for example, determine the current location or allow their determination via a 
combination of such orientation data with data obtained from the position sensors 
are known. In a typical apparatus, the orientation data comprise map information 
that are used for position determination in conjunction with signals supplied by 

10 the position sensors. The establishment of a correlation or a dependency, e.g. 
when such position information is obtained or presented, between the signals 
captured from the eye or light captured from the field of view and the provision of 
information, however, far exceeds that known to the art. 

Figures 1 2A to 1 2E show the field of view 1 290 perceived by a user of an 

15 information system designed in form of a guidance system, according to another 
embodiment. The system evaluates the captured field-of-view light with respect 
to the positioning information obtained via a pattern recognition that takes the 
data available for the determined whereabouts into consideration. Orientation 
hints such as characteristic buildings, side streets, or the like that are to be 

20 expected for the determined whereabouts are thereby recognized such that e.g. 
visual or acoustic guidance or identification can be carried out, if necessary. 
These operations may be performed by the information unit. 

In the illustrated example in accordance with Figure 12A, the guidance 
system serves for navigation. In this case, it is determined, e.g. based on a 

25 computed or predetermined route, available map information and the current 
whereabouts, that one should turn into the second street on the right side. This 
street is recognized on the basis of the captured field-of-view light via pattern 
recognition, in response to which a hinting arrow pointing at the street is 
locationally correctly blended into the field of view via projection taking into 

30 consideration the gaze direcfion determined by the system. Similarly, the 
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guidance system could provide the driver with an acoustic message, e.g. "turn 
right after 50 meters" or "now turn right." 

In the example illustrated in Figures 12B and 12C, the guidance system 
serves to provide information. For example, a user can selectively be provided 

5 information about his direct surroundings. Referring to Figure 12B, a tourist 
using the information system looks at a characteristic building and actuates an 
activation button that is physically present or virtually blended into the field of 
view. The building is subsequently identified on the basis of the determined 
whereabouts and a pattern recognition based on the captured field-of-view light 

10 or an electronic compass that determines the direction of the head. In response, 
information about the building is provided. This information can originate from a 
data bank or other information source and can be selected, e.g. interactively, via 
context-dependent menu that visually or acoustically lists the information 
available for selection for that particular building. The selection can be carried 

15 out via voice control or via fixation with the eyes. Further information re eye- 
controlled menu selection will be described in a later section of this description. 

Referring to Figure 12B, historic data are blended into the field of view via 
projection. In doing so, the system determines, from the captured field-of-view 
light, a suitable blend-in position, e.g. in front of a monotonous roof or in front of 

20 the sky. The data are blended in in accordance with the blend-in position. 
Typically, the fovea centralis is not directed at the blend-in position at first, 
whence the blended-in data is first perceived as an unfocussed, peripheral 
appearance. The locationally fixed, blended-in data are not imaged upon the 
fovea centralis until a corresponding pivoting of the gaze direction in accordance 

25 with Figure 12C. If the gaze is directed at another building recognized by the 
system, then the blend-in information can change in accordance with Figures 
12D and 12E. In the Figures, the circle 1290 represents the perceived field of 
view, whereas the circle 1291 designates the region of the field of view captured 
by the fovea centralis. 
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By using a compact and portable design shown in Figure 1, the 
information system 100 can be used as an orientation system worn by a 
pedestrian, a bicyclist, a motorcyclist or other vehicle driver. 

Figures 13A and 13B illustrate the use of the information system 100 as a 
5 driving aid, in accordance with another embodiment. Preferably, the information 
system 100 includes an information unit comprising a distance sensor (e.g. an 
optical or acoustic distance measuring device) or a radar apparatus. 
Alternatively, the information unit is connected to an external distance measuring 
system. The distance sensor determines the distance between a vehicle and 
10 objects located in front of the vehicle in the direction of motion. In the case of a 
stereoscopic capturing of light from the field of view, the distance could be 
detemriined via a computation of parallax in which change of position of the 
object in a respectively captured left and right image conveys infonnation re the 
distance. 

15 The information unit may also include an evaluation apparatus that can 

determine a (or calculate a probability), that the vehicle is on a collision course 
with an object within the field of view. A user of the system has a perceived field 
of view 1390 (shown in Figs. 13A and 138). Upon determining a potential 
collision course, the driving aid displays, e.g., a warning symbol 1395, which can 

20 be blended in the region of keenest sight 1391 and a warning circle 1394 can be 
blended in around the dangerous object by projection, as described above. If the 
object is located outside or on the edge of the region of peripheral vision, then a 
further warning symbol 1395a can attention to where the danger lurks, as shown 
in Figure 13A. 

25 Other infomnation relevant to driving safety can also be determined via 

sensors or the captured field-of-view light For example, an evaluation apparatus 
could recognize the road lane markings of a road lane lying within the field of 
view via pattern recognition and compute the highest allowable speed, in 
particular in curves, therefrom. If the information system determines, 

30 independently or via connection to the instrument system of a vehicle, that the 
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vehicle has exceeded this computed highest speed, then a warning symbol 1395 
can be blended in in the region of keenest vision 1391 . This is illustrated in 
Figure 13B. The advantage of blending In of a warning symbol 1395 in the 
region of keenest vision 1 391 lies in the fact that the symbol 1 395 appears where 

5 the eye is looking and thus does not tempt the eye to look away from the present 
scene. For this reason, the brightness of blended in symbols should be chosen 
such that the symbol appears translucent. Optionally, the driving aid system can 
also acoustically warn a user about the danger. 

The information system 100 may be a complex, multifaceted information 

10 system. Figures 14A and 14B illustrate the use of the information system 100 as 
a mobile fire department command center, according to another embodiment. 
The mobile fire department command center 1410 includes a command console 
1412 and several helmet systems 141 1 . Each of the helmet systems 141 1 
comprises a signal capturing apparatus, as described above, and a field-of-view 

15 capturing apparatus. Each of the helmet systems 141 1 can optionally be 

equipped with a projection apparatus, infrared sensors or position sensors. The 
helmet systems 141 1 can also be equipped with further sensors that allow, e.g., 
an assessment of the air quality or wind speed. For the purpose of 
communication, each of the helmets 1411 is equipped, for example, with a radio 

20 transmission unit that communicates with the command center 141 0 and/or the 
command console 1412. The radio transmission unit can take over both tasks of 
an information unit as well as tasks of an output apparatus by transmitting and 
receiving information. 

Preferably, the field-of-view images captured by the respective helmets 

25 141 1 which can be brought into agreement with the tmly perceived field of view 
of the respective firemen on the basis of the signals captured from the eyes are 
transferred to the command console 1412 and presented there on monitors. In 
order to reduce the amount of data to be transmitted, users of the command 
console 1412 can also wear a projecting spectacle system so that solely the 

30 image data falling onto the region of the user's fovea centralis must be captured 
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and/or transmitted in high resolution. A correlated field-of-view image of an 
individual fireman or a mosaic of several images could be projected into the 
user's eye. Thus, the user could see exactly that which the fireman sees or be 
provided an image from the fireman's field of vision that changes depending on 

5 his own eye movements. 

In the optional case of a projection, additional infonnation could be woven 
into the image projected to the user and/or the fireman. For example, orientation 
and/or temperature information obtained via the position sensors and/or infrared 
sensors could be blended into the field of view. Constantly blending in particular 

10 points on a compass such as North and West as well as altitude information 
could be helpful reference information both to the user far away from the action 
he is seeing as well as to the fireman veiled in smoke and haze. 

Through appropriate pre-processing of the captured position Infonnation 
and due to the inherent networking of the system components, the position of his 

15 colleagues, e.g. via a characterizing "X," or the position and severity of the 
sighted or otherwise captured hearts of the fire, e.g. via a dot that is colored in 
accordance with the strength of the fire, could be blended in to each fireman. 
This would make fighting the fire easier and would reduce the probability of 
accidentally injuring a colleague hidden behind smoke or a wall. 

20 

Figure 15 shows schematically an information system adapted for the 
operation of an external system, e.g. a remote controlled robot 1579 designed to 
move dangerous objects. A movable robot 1570 includes a camera apparatus 
1571 as well as a grasp arm 1572. The robot 1570 is connected to a spectacle 
25 system 1520 worn by a user 1502 e.g. via a radio connection. The images 
captured mono- or stereoscopically via the camera apparatus 1571 can be 
mono/stereoscopically projected onto the retina of the user 1502 via a projection 
apparatus comprised by the spectacle system 1520. In the case of a 
stereoscopic projection, spatial vision would be ensured. 
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If the camera apparatus 1571 is provided with a macroscopic lens having 
a wider "field of view" than the field of view of the user 1 502, then the field of 
view seen by the user 1502 can be held in correlation with the remote image as a 
function of the captured eye movements of the user 1502 via a corresponding 
5 selection of an image detail from the image supplied by the camera apparatus 
1571 as described above. The movements of the head of the user 1502 can also 
be captured via position sensors such that the camera apparatus 1571 pivots in 
correlation with the head movements. The information system in accordance 
with the invention thus offers a previously unachieved degree of visual 
10 authenticity during the perception of a remote scene, which considerably eases 
the control of such an external system 1570. 

By attaching a microphone to the external system, in particular a 
directional microphone that is directed as a function of the head position or the 
gaze direction, in connection with the headphone arrangement on the spectacle 
15 system allows a further sensory dimension to be realized. 

In order to allow further operational control of the robot 1570, a manually 
operable joystick 1525 is connected to the spectacle system 1520 e.g. via a 
cable 1526. Thus would allow, for instance, the grasp arm 1572 or the direction 
of motion of the robot 1570 to be controlled in several directions. 

20 

Figure 16 schematically illustrates another embodiment of the information 
system using a spectacle system 1620, which acts as a universal remote control 
for one or more devices, for instance a computer, a video recorder 1676, a 
printer 1677, a slide projector and/or a telephone 1679. The spectacle system 

25 1620 provides an interface that communicates in two directions between a user 
1602 and any device 1675-1679 to be controlled. First, the device 1675 through 
1679 must be recognized. This is fundamentally carried out, by gazing at the 
device (1675-1679) to be operated with the fovea centralis. The identity of the 
gazed-at device (1675-1679) can be determined either with or without the 

30 assistance of the device (1675-1679). In the following, it is assumed that both 
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the device (1675-1679) as well as the spectacles 1820 are equipped with the 
signal reception and/or transmission apparatus necessary for the operations 
described. 

If the identity Is determined with the aid of the device 1675-1679, then this 
5 device 1675-1679 either radiates an ID-signal e.g. an infrared or ultrasonic 
signal, In more or less regular intervals or It is requested by a request signal 
radiated by the spectacles 1620 to radiate an ID-signal. The request signal must 
be radiated localized to the gaze direction in order to avoid addressing other 
devices. The ID-signal radiated by the device 1675-1679 Is recognized by the 
10 spectacles, as a result of which conclusions are made re the identity of the 
device. 

If the identity is determined without the aid of the device 1675-1697, then 
the spectacles 1620 carry out a pattern recognition of the gazed-at region of the 
field of view In cooperation with a databank or other information source 1640 that 
15 contains pattern recognition data for the respectively addressable devices 1675- 
1679. 

Based on the identity of the device 1675-1679, a menu that is adapted to 
the possible functions of the device Is blended in, at a fixed location, into the field 
of view of the user 1602, if necessary upon the pressing of a button or the 

20 blinking of an eye. If the functionality of the spectacles is not readily known, then 
the corresponding information is first established from a databank or other 
infomriation source 1640, e.g. via standardized interrogation of the device itself. 
In this case, identification of the device embedded Into the interrogation signal 
ensures that solely the desired device responds to the inten-ogation. By blending 

25 in the menu into the field of view at a fixed location, the user 1602 can control the 
menu, which may be hierarchical if necessary, via slight eye movements like a 
computer menu. 

After the desired function has been selected, a signal corresponding to the 
function is sent from the spectacles 1620 to the device 1675-1679. In this case, 
30 identification of the device embedded into the signal can ensure that solely the 
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desired device reacts to tlie signal. In this manner, quick and easy operation of 
many devices can be achieved with little hardware. 

Figure 17 shows schematically an optical device with a hinged mirror 1755 
allows a switching between a capturing from the field of view and a capturing 

5 from the eye 1 780 or a projection onto the retina 1 781 . The advantage of this 
optical device lies in that the same swiveling mirrors 1754H and 1754V can be 
used for a capturing from the field of view and for a projection onto the retina 
1781 and that the beam path for a capturing from the field of view and the beam 
path for a capturing from the eye 1780 / a projection onto the retina 1781 is, to a 

10 large degree, accordingly identical. In this manner, a high correlation between 
the light captured from the field of view and the signals captured from the eye / a 
high correlation between the light captured from the field of view and the image 
projected onto the retina is achieved through the optical system itself. This 
means that no additional correlation errors are caused by the aforementioned 

15 beam paths traveling across different swiveling mirrors that could exhibit different 
rotation characteristics. For capturing light from the field of view and capturing 
light from the eye, even the same light capturing apparatus 1751 can be used. 
The correlation can solely be negatively influenced by the reflection on the 
spectacle lens 1724 and the optical system of the eye 1780. 

20 

There are other embodiments directed to the use instead of a traditional 
TV devices, newspapers, books, as helmets, diagnostic or treatment devices and 
other applications. 

Previous electronic books and/or newspapers have the disadvantage of 

25 being too heavy and/or too unwieldy and can moreover only present a limited 
amount of information per page. Portable video and TV devices are also heavy 
and/or unwieldy. If the infomiation system in accordance with the invention is 
designed such that the provision of information comprises a projection of image 
information into the eye, then various vision-related media, e.g. electronic books 

30 or newspapers, television or video games, can be implemented via the 
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information system. In such case, the information system in accordance with the 
invention is implemented, for example, in the form of wearable spectacles as 
described above that can be connected e.g. to an information network, a portable 
storage apparatus, e.g. a CD-ROM or DVD-reading device or another information 
5 source via a cable, infrared or radio connection. 

An advantage of such a design of the information system in accordance 
with the invention is that its capturing of signals from the eye in conjunction with a 
field-of-view capturing allows a projection in which the projected text and/or the 
projected images appear to be fixed in space. For this purpose, the information 
10 apparatus comprises an evaluation apparatus that determines the correlation of 
the visual axis to the field of vision and that accordingly controls the projection 
such that the information projected onto the eye appears to be immovable vis-a- 
vis the field of vision in spite of movements of the eye. The determining of the 
correlation of the visual axis to the surroundings can also be assisted by position 
15 sensors mounted in the spectacles. 

The virtual position of the fixation fixed e.g. via a fixation with the eye in 
conjunction with a blinking of the eyes or a pressing of a button or even 
automatically, for example, by using an image-processing evaluation of the field 
of vision that detemnines an area of the field of vision having as little content as 
20 possible. The disruptive effect of the natural field of view not necessarily covered 
up by the projection of information can be reduced via a complementary-color 
"wiping out" in which complementary-colored pixels are determined on the basis 
of the light captured from the field of view whose correlated projection onto the 
respectively associated regions of the retina make the natural background 
25 appear white on account of the addition of colors. If a black background is 
desired, then the perceived total brightness of the projection must exceed the 
perceived total brightness of the natural field of view by roughly 10% to 20% as 
described above so that even the brightest points of the natural field of view are 
perceived as black. 
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For the sake of controlling operation, image information representing 
virtual control knobs can be projected into the eye such that they likewise appear 
fixed in the vicinity of the text and/or image in the field of view. The virtual 
infomnation medium could thus be remote controlled, i.e. page tuming, fast 
5 forwarding, rewinding or the like, by gazing at the corresponding control knob 
with the fovea centralis plus pressing a button or blinking an eye. Similarly, 
access to lexical, databanks, etc. could be made possible by gazing at presented 
words or image sections. Instead of control knobs, the information system could 
also be controlled, for example, via menu guidance in which control menus "pop- 
10 up" when a particular region of the image is observed in order to allow an 
ocularly controlled selection from the menu which may be hierarchically 
constructed, if necessary. 

A further advantage of such a design of the information system in 
accordance with the invention is that the amount of data necessary for a 
15 sufficient, momentary presentation is far less than the amount of data that would 
be necessary for a high-resolution presentation of the entire field of view. This is 
due to the fact that the information system has knowledge of the region of 
keenest sight. Thus, only those portions of the projection must be carried out at 
high resolution that regard the region of the fovea centralis. Onto the other 
20 regions of the retina, a projection having a lower pixel density suffices. The 
amount of data necessary for an instantaneous presentation is accordingly 
reduced, which has clear system advantages. In particular, the perceived size of 
the projected image may be arbitrarily chosen without unprocessably large 
amounts of data for presentation of the instantaneous image being the result. 
25 If the projected Image is larger than the field of view, then the current 

visual axis determines the cropping of the Image. The projection is carried out 
such that the current image detail fills the entire active region of the retina. By 
moving the eyes, further sections of the image can be brought into the field of 
view. If the projected image is smaller than the field of view, then projection must 
30 only be carried out onto a limited portion of the retina. If the natural background 
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of the field of view is not blended out, then this changes during movements of the 
eyes. In particular for television or cinema-like presentations of information, a 
projection that fills the field of view exactly is preferred. 

If signals are captured from both eyes of a user, then the projection can be 
carried out stereoscopically, wherein a slightly different image is supplied to each 
eye such that the brain believes to perceives a three dimensional total image. 
This allows an optimal system-human interface e.g. for 3D television, 3D video 
games, 3D CAD applications or other, in particular interactive, 3D applications to 
be realized. Preferably, the infomnation system comprises further control 
elements, for example a joy stick, pedal or steering wheel that allows a 
navigation and/or change of perspective within the presented virtual image or 
other influencing of the presentation of information or of a system connected with 
the infomnation system. As described above, the eye itself can also act as a 
control element. 

By accordingly applying the above measures necessary for the positioning 
of an electronic newspaper at a virtual location, it is likewise possible to project 
the person wearing the infomnation system in accordance with the invention other 
orientation aids onto the retina such as, for example, an artificial horizon. 

The information system 100 may be arranged for ophthalmologlcal 
applications and visual aids. Due to its capturing of signals reflected back from 
the eye, the information system in accordance with the invention is excellently 
suited for embodiment as an ophthalmologlcal system. For example, the 
information system in accordance with the invention can be implemented as a 
positioning system for ophthalmologlcal surgery, in particular for 
ophthalmologlcal laser surgery. The information system can also be used e.g. as 
an ophthalmologlcal diagnostic system, visual aid system and/or visual deficiency 
correction system. 

Most of the structures or organs of the eye are very small in comparison to 
manual movements. Diseases and injuries to these structures / organs often 
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only affect a small, microscopic area. As opposed to many other parts of the 
body, the eyes, however, cannot be fixed, which makes the treatment of possible 
diseases or injuries to the eye particularly difficult. 

Due to the ability of the information system to exact follow movements of 
the eye and provide information with regard to the momentary position of the eye 
even to other systems, these difficulties can be overcome via a therapeutic 
system on the basis of the information system. For example, the therapy system 
can be connected to the information system for the purpose of exchanging 
information in such a manner that the information with regard to the momentary 
position of the eye is provided to the therapy system such that a high-precision, 
automated therapy of the eye can be carried out even when the eye is moving. 

In accordance with another embodiment, a therapeutic laser beam is 
directed via the optical system. A laser treatment of the eye, in particular of the 
retina, can thus be carried out in the same manner as a projection as described 
above. For example, diseased veins in the choroid coat can be stultified in that a 
photosensitive preparation is injected or taken in and that the diseased portions 
of the choroid coat are precisely irradiated for several tens of seconds. Such a 
therapy can be precisely canried out with the aid of the information system. 

In order to be used as a visual aid and/or visual deficiency correction 
system, the output apparatus of the information system comprises a projection 
apparatus that projects the vision-improving image information onto the retina. In 
addition, the information apparatus comprises an evaluation apparatus that 
determines the vision-improving image infomnation on the basis of the light 
captured from the field of view. The vision-improving Image information is 
preferably projected onto the retina in correlation with the signals captured from 
the eye such that the naturally perceived field of view and the projected image 
information are perceived as a unitary image. In extreme cases, the vision- 
improving image information is projected onto the retina such that the otherwise 
naturally perceived field of view is not at all perceived by the eye. As described 
above, the degree of perception of an image projected in this manner in relation 

- 58 - 



to the naturally perceived image can be controlled via the brightness of the 
projected pixels. 

Such an information system allows e.g. vision deficiency correction for 
short or far-sightedness as well as for color blindness to be carried out. During 
the correction of short or far-sightedness, the information system can set to a 
(quasi-)fixed correction, can allow a variable correction, or can automatically, 
dynamically adjust itself to the visual deficiency. The conrection is carried out via 
an adjustable (If need be), optical focusing system within the projection 
apparatus or via image processing measures. Latter can be implemented at low 
system cost. 

Implementations with (quasi-)fixed or variable correction are 
understandable to the person skilled in the art without further explanation due to 
inherent similarity to similar optical systems. An implementation with a dynamic, 
automatic correction of the natural imaging error comprises, in addition to the 
aforementioned correlation, a further dependency on the signals captured from 
the eye. In such case, in particular a retinal reflex image is captured that 
supplies, via comparison with light captured from the field of view and/or via 
image processing evaluation, infomnation re the sharpness of the image imaged 
onto the retina. The light captured from the field of view Is accordingly processed 
into vision improving image information and projected onto the retina. The 
information system can also act as a diagnostic system through output of the 
correction values determined in this manner. 

Due to its capturing of signals reflected back from the eye and light 
originating from the field of view, the information system in accordance with the 
invention is in a position, by using a con-espondlngly programmed evaluation 
apparatus, to supply infomiation about many ophthalmologlcally relevant 
characteristics of the eye. For example, squint angle, primary positions (PP), 
visual field testing even with colors, threshold tests, standardized testing 
methods for glaucoma diagnosis, retinal function tests (e.g. ERG and VEP) even 
at selected locations and tests of the receptive fields can be carried out / 
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determined. The person skilled In the art selects the signals to be captured from 
the eye for this purpose, the field-of-view stimuli necessary for this purpose and 
the processing algorithms necessary for this purpose on the basis of his 
specialized knowledge, accordingly taking into consideration the invention 
5 described above. 

Whereas e.g. the keenness of the vision can be detennined through an 
evaluation of signals reflected back from the eye and can be subsequently 
corrected, the correction of many other visual deficiencies presumes a system- 
independent determination of the deficiency, for example by an ophthalmologist. 
10 A befitting setting of the correction carried out by the information system can be 
carried recursively or simply. 

In a recursive adjustment process, a correction is canied out by the 
information system in accordance with a previous setting while the visual acuity 
of the person with defective vision is being tested. On the basis of the results of 
15 the tests, a new setting of the information system is chosen. This process is 
repeatedly carried out until the visual deficiency has been sufficiently 
compensated. In this manner, the infomnation system acts equally as a 
diagnostic system since the visual deficiency can be detennined based on the 
best-correcting final setting. 
20 In a simple setting processing, the visual acuity of the person with 

defective vision is tested without any type of compensation. Based on the results 
of the tests, a suitable setting of the information system is chosen that then, in 
later operation, prepares the light captured from the field of view into vision- 
Improving image infonnation in accordance with this setting and projects it onto 
25 the retina. During the preparation, e.g. particular spectral components or 

particular regions of the field of view are emphasized or modified through other 
image processing measures in accordance with the setting, i.e. the original visual 
deficiency. 

For people suffering from night blindness, a visual aid can be realized via 
30 the information system in accordance with the invention, for example, in that the 
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light captured from the field of view, e.g. via highly light-sensitive photo-detectors, 
is strongly amplified and projected onto the retina. In this manner, the cones can 
be stimulated in such manner that predominantly color, photopic vision instead of 
scotopic vision takes place. Also, the maximally allowable brightness of the 
5 individually projected pixels is limited to a predetermined threshold value in order 
to avoid a glaring through brightly illuminating objects such as street lamps and 
oncoming cars. Such a system is thus also suitable as an anti-glare system 
since, if the brightness of the entire field of view is raised, whereas the 
"excessive" brightness of individual pixels is left unchanged, then the 

10 "excessively" bright pixels are not perceived as being "excessively" bright. If the 
information apparatus also comprises an infrared sensor that captures infrared 
light from the field of view, then additional, monochrome image information with 
regard to the field of view can be obtained by night or fog that can be 
transformed into the visible spectral range in order to improve the image 

15 information already obtained via the field-of-view capturing apparatus and the 
evaluation apparatus. 

In general, the information system in accordance with the invention can 
also be suitable for improving visual acuity. E.g. in the case of strong or weak 
contrasts or in the case of low brightness in the field of view, image infomnation 

20 that is adjusted with regard to its brightness can be projected into the eye in 
order to allow improved visual acuity. 

The information system 100 may be integrated into a helmet such as a 
fireman's helmet described above. Similar embodiments, e.g. as a soldier's, 

25 driver's, crane operator's, sportsman's or pilot's helmet or spectacles are 

possible. A soldier's helmet / spectacles on the basis of the information system 
in accordance with the invention could be of aid to the soldier, for example, for 
orientation and/or for targeting. In such case, the information apparatus of the 
information system preferably comprises sensors and/or radio receivers that 

30 allow an extrasensory perception of the surroundings and/or the reception of 
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information from a command center. The output apparatus will provide 
information preferably visually, acoustically or tactually, e.g. in tlie form of sfiort 
electric stimulating currents on tlie sl^in. Latter could be used to directly inform a 
soldier of the direction of a foreign object approaching from behind. 
5 As a night vision device, the infomiation system would also capture 

infrared light from the field of view in addition to the capture of visible light from 
the field of view. As described above, image information can be obtained from 
such captured infrared light and be employed in the enhancement of image 
infomnation to be projected into the eye. 
10 If the infomiation apparatus comprises e.g. a GPS receiver, then the 

^ helmet could project position infomiation or orientation aids onto the retina. 

□ Preferably, the projection of such information into the eye is carried out similar to 

^ the projection of an electronic newspaper. This avoids a distraction of the soldier 

W since the information appears to be fixed in space or vis-a-vis a neutral position 

15 of the eye. An adaptation of the image information to the background perceived 
therebehind for the sake of best possible readability also takes place via an 
M= evaluation apparatus belonging to the information apparatus, 

g Although a radio transmission or other data transmission from the soldier 

O to a command center is generally to be avoided from strategic reasons of 

20 camouflage, a transmission of field-of-view data con-elated to the eye movements 
of the soldier to a command center could also be meaningful in particular cases. 

In an embodiment that is particular interesting for soldiers, the information 
apparatus comprises one or more cameras that capture images from outside the 
field of view. The image information obtained in this manner is then projected 
25 onto the retina via a projection apparatus. The supplementary image projected 
onto the field-of-view image could be projected, for example, as an image within 
an image as a small image in the corner of the natural or projected field-of-view 
image or appear as a longitudinal strip along the bottom edge. In this case, the 
capture of signals from the eye serves, together with the capture of the field of 
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view, to maintain the projected images in correlation witli the movements of the 
eye. 

For a crane operator, it is helpful to project in supplementary images from 
other perspectives into the field of view. The infomiation system includes 
5 supplementary sensors providing aid distance or weight information that is 

projected into the field of view. Such information can also be provided audibly or 
visually, e.g., upon gazing at the load in combination with the clicking of a button. 
In this case, the light determined from the field of view serves as a basis for the 
image recognition, whereas the signals from the eye allow a correlation of the 

10 captured field of view to the visual axis as described above. 

The information system 100 can provide a pilot with many various types of 
information. Via a connection to the information system of an airplane, relevant 
data such as flight altitude, speed or direction of flight or even an artificial horizon 
could be blended in to the pilot's field of view, for example, as described above. 

15 During landing, landing aid information could also be blended in that depict a 
virtual landing corridor or indicate altitude or direction correction values. In 
military applications, friend/foe and targeting aid information could be provided to 
the pilot. In this case, the gaze direction of the pilot plays a role both during the 
spatial blending in of the information as well as during information selection. The 

20 pilot would like a flying object upon which he has fixed his eyes' gaze to be 
identified. If the identification is carried out visually, he does not want the 
blending in to cover any relevant areas of his field of view. In this case, due 
consideration must be given to the contrary requirements that the relevant 
regions of the field of view are typically imaged onto the fovea centralis but also 

25 that only those images that are projected onto the fovea centralis are sharply 
imaged. Thus, an intelligent blending in must be carried out in which the relevant 
regions of the field of view are recognized, for example, via image recognition 
and not solely via the orientation of the fovea centralis. In this respect, the 
information system in accordance with the invention can also act as a sub- 

30 system to the information system of the aircraft and provide information thereto. 
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In this manner, e.g. information with regard to where the pilot is looking could be 
supplied to the aircraft information system by the information system in 
accordance with the invention and contribute there to target capturing. In true 
action, the information system could locate enemy radar positions via sensors 
5 and depict their position together with the associated landscape in three 
dimensions. 

Various types of infonnation could be provided to sportsmen via the 
information system in accordance with the invention as in the examples above. 
Orientation aids, speed information and/or enhanced field-of-view information 

10 that allows better vision at dusk, at night, in rainy spray or fog could be provided, 
for example, by projecting infomriation into the eye. A non-visual provision of the 
Information is particularly suitable in the case of low-content infonnation. Similar 
to the above examples, an infonnation system in accordance with the invention 
worn by a sportsman could act as a sub-system of a sporting device or of a 

15 vehicle. 

The information system 100 can provide extrasensory perception using 
one or more sensors, e.g. magnetic field detectors, pressure sensors, 
thermometers, spectral sensors, optical or acoustic interference measuring 

20 devices. In particular in the case of a superimposition of a pictorial presentation 
of infonnation obtained from the sensors onto the natural field of view via 
projection into the eye, the presentation corresponds to the needs of the person 
having vision. In such a case, the information system in accordance with the 
invention can appear as a component, in particular as a presentation apparatus, 

25 of a complex measuring apparatus. 

An example of such a system are spectacles equipped with sensitive 
magnetic sensors that is in a position to locate cun-ent-conducting or metallic 
objects in correlation to the spectacles. If such located objects are designated, 
true to their position and in color, in the natural field of view by projection that 

30 enables locating, e.g., water pipes or electric wiring running under plaster. A 
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handyman wearing such a spectacle system would see the path of the piping or 
wiring as "painted on the wall". 

If a two or three dimensional array or other one or multi-dimensional 
distribution of the sensors is chosen, then even e.g. highly complex vector fields 
5 or gradients could be made visible to an observer as an image over the object or 
arrangement associated therewith. For example, an arrangement of pressure 
sensors around a test object in a wind tunnel could supply pressure information 
that is prepared, in such a manner, via the information system in accordance with 
the invention as described above and projected into the eyes of an observer who 

10 Is observing the test object through a window such that he sees the pressure 
gradients resulting from the test object there where they are present based on 
appropriate, colored depiction of the pressure values. Temperature information 
obtained using an infrared camera could be presented to a welder in his field of 
view such that the local surface temperature along the work piece is 

15 recognizable. 

Similarly, spectral sensors could be used to give a user information about 
exact color values or material compositions. In this case, it is also practical to 
present the detennined information audibly depending on exactly where the user 
is looking. In conjunction with a databank and pattern recognition, such a system 

20 could be used, for example, to at least approximately identify mushrooms or 
plants, wherein the user, upon system request, looks at particular parts of the 
mushroom / plant and/or turns these to face the sensors. 

Having described various embodiments and implementations of the present 
invention, it should be apparent to those skilled in the relevant art that the foregoing 

25 is illustrative only and not limiting, having been presented by way of example only. 
There are other embodiments or elements suitable for the above-described 
embodiments, described in the above-listed publications, all of which are 
incorporated by referece as if fully reproduced herein. The functions of any one 
element may be carried out in various ways in alternative embodiments. Also, the 
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functions of several elements may, in alternative embodiments, be carried out by 
fewer, or a single, element. 
What is claimed is: 



D 
D 
01 

m 
m 
m 

ru 

5 

Q 

y 

o 

s 



- 66 - 



